Abstract
This report analyses 1,243,902 survey responses collected on the Tayqun platform between January 2025 and April 2026 across 4,331 studies fielded primarily in the MENA region. We measure the prevalence of seven fraud and inattention signals, quantify their co-occurrence, and estimate the improvement in data reliability from a layered 14-check validation pipeline. Headline result: 12.4% of unfiltered responses carried at least one quality flag, and filtering them improved multi-item scale reliability (Cronbach's α) from 0.71 to 0.83 on the median study.
Key findings
- 12.4% of unfiltered responses contained at least one quality flag - consistent with the 10-15% range reported across the panel industry
- Speed and straight-lining together accounted for 71% of all flags, making them the highest-yield checks per unit of implementation effort
- Flags cluster: a response with one flag had a 42% chance of carrying a second, versus a 12.4% base rate - low quality is a respondent state, not a random event
- Studies with attention checks enabled saw 38% fewer low-quality responses survive to the final dataset
- Trust-score gating below 30 removed only 4.1% of responses while preserving 96% of net-promoter signal - high precision, low collateral damage
Method
Every response was scored on a 0-100 quality scale combining 14 weighted checks: completion speed against estimated time, attention-check performance, straight-line detection across rating items, answer-consistency validation, duplicate-IP detection, open-text quality (length, gibberish, character repetition), selection-pattern analysis, geo-validation, per-question timing, device fingerprinting, statistical outlier detection, historical trust score, and manual review outcomes. We compared researcher acceptance decisions against automated scores, and measured scale reliability on the 612 studies containing at least one multi-item scale.
Prevalence by signal
| Signal | Share of flagged responses | Share of all responses |
| Speeding (<20% of estimated time) | 44% | 5.5% |
| Straight-lining (>85% identical ratings) | 27% | 3.3% |
| Failed attention check | 14% | 1.7% |
| Low-quality open text | 8% | 1.0% |
| Duplicate IP (3+ completions) | 4% | 0.5% |
| Geo-mismatch | 2% | 0.2% |
| Statistical outliers | 1% | 0.1% |
The case for layering
No single check is sufficient. Speed detection alone catches the laziest cases but misses the "slow straight-liner" who leaves a tab open while clicking one column. Straight-line detection misses randomizers, who are in turn caught by consistency validation and attention checks. In our data, the 14-check portfolio flagged 2.9× as many problem responses as the single best check alone, at a false-positive rate (measured against human reviewer panels) of 1.8%.
“Quality controls are a portfolio, not a silver bullet.”
- From the report's conclusion
Impact on research outcomes
On the median multi-scale study, filtering flagged responses moved Cronbach's α from 0.71 to 0.83 - the difference between a scale most reviewers would question and one they would accept. Effect-size estimates in A/B concept tests shifted by a median of 2.1 percentage points after filtering; in 6% of studies, filtering changed which concept won. Bad data does not just add noise; occasionally it changes the answer.
Limitations
All studies were fielded on a single platform with non-probability recruiting concentrated in the MENA region. Flag thresholds are the platform's own; other panels using different thresholds will report different prevalence. Findings should be read as directional benchmarks rather than population estimates.
Companion studies drill into individual checks: straight-lining thresholds, attention-check placement, and the predictive power of trust scores.