The two folk theories
Researchers hold two contradictory beliefs about incentives. The first: higher rewards attract professional survey-takers and fraud, degrading quality. The second: higher rewards signal respect and buy effort, improving quality. Both are stated as obvious. They cannot both be true, and pricing decisions on the platform depend on which one the data supports.
Design
We fielded an identical 10-minute consumer survey to 4,800 participants across four randomized reward arms: $0.75, $1.50, $3.00, and $6.00. Participants saw only their own arm's reward. We measured fieldwork speed, response quality (composite 0-100 score), fraud-signal prevalence, open-text effort, and the demographic composition of completes. All arms drew from the same eligible pool with identical targeting.
Results
| Reward | Time to fill quota | Mean quality score | Fraud-flag rate | Median open-text length |
| $0.75 | 68 hours | 76.8 | 3.1% | 34 chars |
| $1.50 | 31 hours | 78.4 | 2.9% | 39 chars |
| $3.00 | 9 hours | 79.1 | 3.0% | 44 chars |
| $6.00 | 4 hours | 79.4 | 3.3% | 46 chars |
What moved and what did not
- •Speed moved enormously. Doubling the reward from $1.50 to $3.00 filled the quota 3.4× faster. If fieldwork deadlines matter, incentives are the lever.
- •Quality moved a little, upward. +2.6 points from the lowest to highest arm, driven almost entirely by open-text effort (+35% length, higher informativeness ratings). Ratings-based measures barely changed.
- •Fraud did not move. Flag rates were statistically flat across arms. The fraud-magnet theory found no support - plausibly because quality gates make fraud unprofitable at any reward level, so the marginal fraudster does not appear when pay rises.
- •Composition moved subtly. The $0.75 arm skewed toward high-frequency responders (participants completing 20+ surveys/month); higher arms pulled in more occasional participants, mildly improving demographic balance.
Interpretation
Neither folk theory survives contact with the data in strong form. Higher rewards do not poison the well, and they do not transform data quality either - on a platform with response validation, quality is protected by the gates, not the price. What the reward actually buys is speed, open-text effort, and access to respondents who are not survey regulars.
“Pay does not buy honesty - the quality system enforces that. Pay buys speed, effort, and people you could not otherwise reach.”
- From the discussion section
Practical guidance
- Price for your deadline: quota-fill time is hyper-sensitive to reward, quality is not
- If open-text answers are the point of your study, pay above the platform median - it is the one quality dimension money reliably improves
- Do not underpay to "filter for intrinsically motivated respondents" - what you actually select for is professional speed-runners
- Fair pay also compounds platform-wide: it retains the casual, demographically balanced participants every researcher needs
Limitations
One survey topic, one region, four price points. Effects may differ for B2B audiences, very long instruments, or rewards far outside the tested range. The fraud finding is conditional on active quality gating - platforms without validation may well see the fraud-magnet effect we did not.