Synthetic research: what to trust, and when to add real people
AI estimates landed about four points from real survey results on average. That's close enough for a first look, and not close enough for every decision.
By Jason Karpeles Senior Principal, Data Science 3 min read
Teams need to understand an audience before they commit to a product, message, or campaign. Surveys answer that, but recruiting takes time and money, so early decisions often go ahead on thin evidence. Synthetic research offers another starting point. The hard part is knowing which outputs deserve confidence.
Figure 1
Thirty real people plus AI beat either one alone
Average gap from the full survey results, Federal Reserve test, in percentage points (lower is closer)
Bar chart: 30 real respondents alone were 5.8 points off, AI alone 3.6, 30 real respondents plus AI 3.2, and 300 real respondents alone about 1.7
Close on average, looser on any single answer
The estimates averaged 3.6 and 4.2 points from the real results. Those are averages. Individual estimates could be substantially less accurate.
Two details about method mattered.
Figure 2
Asking about each group beats assuming one answer for everyone
Average gap from the real result, in percentage points (lower is closer)
Bar chart: assuming every group answered like the overall population was 6.9 points off, asking about each age group was 4.0; direct population estimates were 4.7 points off, role-playing respondents with demographic profiles was 8.3
That second result matches published work. A 2024 study by Bisbee and colleagues found that responses from a model prompted with demographic personas came close to survey averages but varied less than real respondents, and shifted with small changes in wording and over time (Political Analysis).
A check on audience fit, not a ranking
Personas matched to the target audience showed higher purchase intent for 22 of 25 concepts. That supports using AI to explore whether an idea suits its intended audience.
Figure 3
Target-matched personas liked the concepts made for them
Concepts where personas matched to the target audience expressed higher purchase intent than mismatched personas
22 of 25 concepts showed higher purchase intent from target-matched personas
It doesn't establish reliable concept rankings. When real results set the top and bottom concepts, AI explanations are useful hypotheses for refinement. When AI sets both the ranking and the reasons, the explanations describe its own preferences.
When real results set the top and bottom concepts, AI explanations are hypotheses for refinement; when AI sets both the ranking and the reasons, the explanations describe its own preferences
Thirty real people, plus the AI
A small real sample improves the result. In the Federal Reserve test, 30 real respondents combined with AI came closest of the low-cost options, at 3.2 points.
The cost advantage comes from recruiting fewer people. At an illustrative $20 per completed response, 300 respondents cost $6,000 and 30 cost $600. Total savings are smaller once design, AI, and analysis are counted.
Fewer paid responses, 30 respondents instead of 300
Federal Reserve test
Respondent cost, at an illustrative $20 per completed response
Illustrative
Saved on respondent costs, before design, AI, and analysis
Illustrative
A rule for when to add real people
Ask one question: would a four-point miss change the decision?
No. Start synthetic.
Early exploration, narrowing a long list, or checking whether an idea fits its audience.
Maybe. Add a small real sample.
Choosing between two close options.
Yes. Run the larger study.
Pricing, a launch decision, or anything you'll have to defend to finance.
Where Alli fits
Two Alli products sit in this territory, so here is how these findings apply to them. Alli Audience Planner builds Live Personas from behavior data, not from a model role-playing a demographic profile. Alli Creative Labs scores creative before launch.
Both are meant as the first look this article describes, with real results brought in where the decision needs more confidence.
Planning research for a launch? Start with a persona, then size the real sample.
Frequently asked questions
How accurate is synthetic research?
In the tests here, AI estimates averaged 3.6 points from the real result on 60 Federal Reserve questions and 4.2 points on 32 Pew questions. Those are averages, and individual answers could be much further off.
Should an AI model role-play survey respondents?
In these tests, no. The same model role-playing respondents with demographic profiles was 8.3 points off, against 4.7 for direct population estimates. Published work also finds persona-prompted responses vary less than real people and shift with wording.
Is a small real sample plus AI as good as a full survey?
No. 30 real respondents plus AI averaged 3.2 points from the full results, better than AI alone at 3.6, but 300 real respondents averaged about 1.7. It's cheaper, not equivalent.
When should you add real respondents to synthetic research?
Ask whether a four-point miss would change the decision. If not, start synthetic. If it might, add a small real sample. If it would, such as pricing or a launch, run the larger study.
Sources
- Political Analysis, Synthetic Replacements for Human Survey Data? The Perils of Large Language Models, James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson, 2024, 32(4), 401 to 416.
- arXiv, How Many Human Survey Respondents is a Large Language Model Worth? An Uncertainty Quantification Perspective, Chengpiao Huang, Yuhang Wu, and Kaizheng Wang. Further reading.
Last updated November 10, 2026.