Attention Checks: Cost, Benefit, and Best Placement

Question

Attention checks are the most direct quality instrument a researcher controls - but they are not free. Each one lengthens the survey, interrupts the respondent's flow, and signals distrust. How many should a survey include, and where should they sit?

Experiment

We fielded the same 12-minute consumer survey to 6,000 participants, randomized into four arms: zero, one, two, or four attention checks. Placement in the four-check arm was at 20%, 40%, 60%, and 80% of survey progress; the two-check arm placed them at 40% and 60%. Low-effort responding was measured independently of the checks themselves, using speed, straight-lining, and open-text quality signals, so the arms are comparable on ground truth.

Results

The diminishing-returns cliff

The move from one check to two buys 24 points of detection for essentially nothing. The move from two to four buys three points - and pays for them with a 3.2-point increase in abandonment and open-text answers 14% shorter across the whole survey. The shortened answers are the more damaging cost: they come from attentive respondents, who apparently read the repeated checks as a signal that the survey values compliance over thought.

“The third and fourth checks do not catch cheaters. They discourage the honest.”

- From the results discussion

Placement findings

Within the two-check arm, checks in the middle third of the survey (40-60% progress) outperformed early placement. Early checks (first 20%) missed responders whose attention decayed later - the most common decay pattern - while late checks flagged respondents already too far in to screen out cheaply.

Recommendation

Two checks in the middle third is the platform default for surveys longer than five minutes; shorter surveys get one. Researchers can adjust this in survey settings, but should treat four or more checks as a data-quality risk rather than a data-quality tool. For the wider system these checks sit in, see the annual quality report.