Most bad surveys were doomed before the first response arrived. The questions were ambiguous, the audience was wrong, or the data could never have answered the question the team actually had. This checklist front-loads the thinking so your budget buys answers, not regrets.
Before you write a single question
- •Write down the decision the data will inform. "We will launch in market X if intent-to-buy exceeds Y%" is a study; "let's see what people think" is a fishing trip.
- •Define the audience in one sentence. If you need two sentences, you need two surveys.
- •Estimate the sample size before setting the budget - our sample size guide covers the actual math.
Question design
- •Keep it under 8 minutes - completion quality measurably degrades beyond that
- •One idea per question. "Was the product good and affordable?" is two questions wearing one question's clothes.
- •Use balanced scales with a neutral midpoint, and keep the same direction throughout
- •Randomize option order on brand and preference lists to cancel position bias
- •Put demographics at the end - opening with age and income feels like a form, not a conversation
- •Pick your rating scale deliberately - see 5-point vs 7-point Likert scales
Targeting and quotas
Tighter targeting raises your effective cost per response and slows fieldwork. Start broader than feels comfortable, watch the first 50 responses, then narrow. It is much cheaper to filter in analysis than to re-field a study that closed with an empty quota cell.
Quality settings
Leave speed, straight-line, attention-check, and duplicate-IP detection on for every study - they have near-zero false-positive cost. Add outlier and pattern detection for numeric and Likert-heavy instruments.
Attention checks: two, not five
Our experiments show two attention checks catch 78% of low-effort responders. A fourth check catches almost nobody new but increases drop-off by 3.2 points and shortens open-text answers.
The final five minutes
- Preview the survey on a phone - most responses in MENA come from mobile
- Take it yourself at a natural pace and check the estimated time is honest
- Have a colleague take it cold, with no explanation from you
- Check every routing path leads to an end screen
- Confirm the reward is fair for the time asked - underpaid surveys select for exactly the respondents you do not want