Are synthetic respondents accurate? What the evidence says, both ways

Every buyer’s first question is whether simulated people track real ones. Vendors answer with a correlation from their own study. Critics answer with a study of prompt-based personas that failed. Both are describing real results from different systems. Here is the record, with the sources attached.

The headline claims

Vendor and analyst content in 2026 repeats a band: 85 to 95% agreement with human samples on structured tasks such as ranking, pricing and sentiment. The band has a real origin, described below, but it is quoted far beyond the systems and questions it was measured on. One vendor’s own buyer guide, published in July 2026, put the corrective plainly: “Do not rely on universal accuracy numbers or vendor correlation metrics. Accuracy varies significantly across target populations, subject complexity, prompt design, and evaluation benchmarks.”

Where the evidence is favourable

The Stanford result is the strongest positive finding in the field, and its design is the point. The agents were not told to “act as a 45-year-old accountant”. Each was given the full transcript of a two-hour interview with one real person, life story and views included, and asked to answer as that person. Accuracy was normalised against how consistently the same people answered two weeks apart, which is the right yardstick: no simulation can beat human test-retest reliability, so 85% of it is a meaningful ceiling.

A second favourable result comes from consumer research. Maier and colleagues tested a method they call semantic similarity rating on 57 personal care product surveys with 9,300 human responses and reported 90% of human test-retest reliability with realistic response distributions. The method matters: instead of asking the model for a number on a scale, it elicits a textual answer and maps it to a Likert distribution by embedding similarity, which avoids the flattening that direct rating produces.

Nielsen Norman Group’s August 2025 review of three simulation studies adds texture. The effect sizes in the Stanford interview-based twins showed a near-perfect correlation (r = 0.98) with those in the human data; digital twins trained on General Social Survey data reached 78% accuracy when backfilling missing answers but 67% when predicting answers to entirely new questions; and synthetic data in a marketing study followed the human trends but differed significantly in magnitude, with a standard deviation consistently lower than the human data.

Where it is not

The most cited negative result is Bisbee, Clinton, Dorff, Kenkel and Larson in Political Analysis, 2024. They prompted ChatGPT to adopt demographic personas and rate eleven sociopolitical groups on feeling thermometers, then compared the output with the 2016 to 2020 American National Election Study. Averages matched closely. Everything else did not: there was less variation than in the real surveys, regression coefficients often differed significantly, the distribution of answers varied with minor changes in prompt wording, and the same prompt gave significantly different results over a three-month period. Their conclusion raises “serious concerns about the quality, reliability, and reproducibility of synthetic survey data generated by LLMs.”

Notice what kind of persona that was: a prompt with demographic and political attributes, no grounding in any real person’s words. It is the first of the four kinds of system, and it is still the most common one sold.

The mechanism behind the flatness has a name. Sharma and colleagues showed in 2023 that assistant models trained on human feedback learn to match a user’s beliefs over the truth, a behaviour the paper calls sycophancy, and that people and reward models both prefer convincingly written agreeable answers a non-negligible fraction of the time. A persona that wants to please is a persona that will not give you the variance of a real segment. The sycophancy note goes into the counters.

Why both are true

The pattern is consistent. Aggregates are the easy part: a language model has read a great deal about most groups and will reproduce the centre of their opinions. Variation, subgroups and novel questions are the hard part, and they are where prompt-based personas fail. Grounding in real material about real people is what moved the Stanford agents from the second row to the first.

The question is not whether synthetic respondents are accurate. It is what this one was built from, on which questions it was tested, and whether anyone compared distributions or only means.

How to read a vendor’s number

• Against what. A correlation against a human dataset the vendor also chose is weaker than one against a public benchmark you can inspect.
• On what population. A result on a mainstream consumer segment says little about niche B2B buyers or hard-to-reach groups.
• Means or distributions. Ask for variance, subgroup differences and the direction of effects, not only averages. Bisbee’s personas matched averages too.
• Which questions. Ranking and stated preference are easier than novel concepts, magnitude of reactions or anything longitudinal.
• Repeated when. Same prompt, three months later. If the vendor has not checked, the number has a shelf life it does not admit.

The validation method turns this list into a procedure you can run yourself in an afternoon, and the vendor questions put it in the room.

What Perplica claims

Nothing numeric, yet. Perplica builds grounded personas from your material, stores the reasoning behind every reply, and compiles reports where each finding cites the turn it came from. Those are design choices that the evidence above says matter. They are not an accuracy claim. No public concordance study of Perplica exists; the first one is the next milestone, and its results will be published in full, failures included. Until then the label reads as it should: a rehearsal, not a replacement, for talking to real customers.