Background
Straight-lining - selecting the same option across most or all rating questions - is among the strongest indicators of inattentive responding. But it is also, occasionally, what honest agreement looks like: a delighted customer really might rate everything a 5. The detection threshold therefore embodies a trade-off, and choosing it by intuition risks either letting fraud through or punishing your most satisfied respondents. We chose it by experiment instead.
Design
We applied five candidate thresholds - flagging responses where 70%, 75%, 85%, 90%, or 95% of rating answers were identical - to 320 studies with at least 12 Likert-type items each (total n = 96,410 responses). Ground truth came from an independent panel of three trained reviewers who classified a stratified sample of 4,800 responses as attentive or inattentive using the full response record (open-text quality, timing, consistency), blind to the automated flags. Inter-reviewer agreement was substantial (κ = 0.79).
Results
| Threshold | Reviewer-flagged cases caught | False-positive rate |
| 0.70 | 97% | 9.2% |
| 0.75 | 95% | 5.6% |
| 0.85 | 91% | 1.4% |
| 0.90 | 78% | 0.6% |
| 0.95 | 62% | 0.2% |
The curve has a clear knee at 0.85: moving from 0.85 to 0.75 buys four points of recall at four times the false-positive rate, while moving to 0.90 sacrifices thirteen points of recall to save less than one point of false positives. False positives at 0.70-0.75 were concentrated exactly where theory predicts - short satisfaction surveys answered by genuinely satisfied customers.
Recommendation
0.85 is now the platform-wide default (the fraudStraightLineThreshold setting). Researchers studying populations with expected uniform responses - brand advocates, employee surveys in high-engagement teams - may relax to 0.90. We recommend against going below 0.85 in any design; if you suspect widespread inattention, add an attention check rather than tightening this threshold, since the two checks fail independently.
Limitations
Reviewer classification, while blind to flags, could see the same signals the other automated checks use, which may inflate agreement. Thresholds were evaluated on surveys with 12+ rating items; on shorter batteries the statistic is noisier and the same threshold behaves more conservatively.
This study is part of the quality-methods series alongside our attention-check placement experiment and the annual quality report.