Motivation
Per-survey quality checks evaluate one response in isolation - they know nothing about who submitted it. A historical trust score inverts this: it compresses every prior response a participant has made into a single number. The question is whether that history actually predicts future behavior, or whether response quality is mostly situational (a bad day, a boring survey) rather than dispositional.
Sample and design
We took the 87,402 participants who had completed at least five surveys on the platform. For each, we held out their most recent response and used the trust score computed before that response as the sole predictor of whether the held-out response would be flagged by the quality pipeline. This is a strict temporal design: the predictor never sees the outcome it predicts.
Findings
- •Trust score alone predicted next-response flagging with an AUC of 0.83 - better than any single per-survey check (the best, speed detection, scores 0.74 on the same task)
- •Participants with a trust score below 30 had a 7.1× higher flag rate on the next response than those above 80
- •Combining trust score with the per-survey check battery lifted AUC to 0.91 - the two layers capture substantially different information
- •Prediction was strongest for chronic low-quality responders and weakest for first-time offenders, exactly as a history-based measure should behave
Quality is dispositional
The size of the effect settles the situational-vs-dispositional question decisively for this population: low-quality responding is overwhelmingly a stable trait of a small minority of accounts, not random noise distributed across everyone. The practical implication is that identifying and gating that minority protects data quality far more efficiently than tightening thresholds on everyone.
“The best predictor of the next response is every response before it.”
- From the discussion section
Operational implications
- The 30-point auto-flag threshold (fraudTrustScoreMin) catches the bulk of repeat low-quality responders while leaving new participants - who start unscored - unaffected
- Researchers should keep both layers enabled: per-survey checks catch first offenses, trust gating catches recidivists
- Because the score is a simple approved/total ratio, it recovers naturally with sustained good behavior - gating is a filter, not a life sentence
Limitations
The outcome variable (flagging) is produced by the same pipeline whose checks we compare against, so per-survey AUCs are conservative estimates. Participants with fewer than five completions were excluded; cold-start prediction is a separate problem. For how the score works day-to-day, see Understanding Your Trust Score.