Scientific guide / v1.0
How to evaluate synthetic personas.
A synthetic respondent is only valid for a named claim, population, task, model version, and error tolerance. This framework makes those boundaries testable.
Start by naming what “works” means.
Believable language, aggregate resemblance, subgroup inference, and individual prediction are different claims. Evidence for one does not validate the others.
Generate hypotheses, stimuli, or edge cases.
Human review of usefulness and novelty; do not infer population prevalence.
Approximate a response distribution for a defined population and task.
Held-out human distribution, uncertainty, variance, calibration, and rare-response coverage.
Estimate differences between prespecified groups.
Adequate human samples per subgroup, subgroup error, and multiplicity-aware uncertainty.
Predict a particular person’s response or behaviour.
Held-out individual outcomes, self-retest ceiling, calibration, and explicit failure cost.
Six steps from claim to decision.
- 01
Define the decision
State the exact claim, user, downstream decision, acceptable error, and cost of a false conclusion before generating synthetic data.
- 02
Freeze a human reference
Use human responses or behaviour that were not used to build, retrieve for, tune, or select the synthetic personas. Record sampling frame, field dates, exclusions, and missingness.
- 03
Lock the system
Record provider, model, prompt, retrieval sources, persona construction, temperature, tools, seed where available, and run date. Archive the executable configuration.
- 04
Measure the distribution
Compare more than means: variance, correlations, calibration, subgroup error, tails, rare responses, and uncertainty must match the intended decision.
- 05
Repeat and perturb
Repeat runs and test prompt, date, and model-version changes. Report run-to-run variance and whether conclusions cross the preregistered decision threshold.
- 06
Escalate to humans
Require fresh human research when the decision is consequential, novel, heterogeneous, weakly calibrated, or outside the validated population and task.
Publish a minimum evaluation record.
A reviewer should be able to reconstruct what was tested, against whom, with which system, and why the reported error was acceptable.
Name the claim
Separate ideation quality, aggregate distribution fit, subgroup inference, and individual prediction.
Required record: The intended decision and failure cost are stated before outputs are inspected.
Hold out humans
Use human responses or behaviour that were not used to construct, retrieve for, or tune the personas.
Required record: Reference data, sampling frame, dates, exclusions, and leakage controls are disclosed.
Test distributions
Report variance, calibration, subgroup error, correlations, and rare-response coverage—not mean agreement alone.
Required record: The metric matches the downstream decision and includes uncertainty intervals.
Repeat the system
Repeat prompts, runs, dates, and model versions; lock configurations needed for reproduction.
Required record: Run-to-run and version drift stay within a preregistered tolerance.
Probe failure modes
Check stereotype amplification, persona collapse, refusals, contamination, missing minorities, and prompt sensitivity.
Required record: Failures are reported by subgroup and cannot be averaged away by overall fit.
Escalate to people
Define when synthetic evidence is exploratory and when fresh human research remains mandatory.
Required record: High-consequence, novel, heterogeneous, or weakly calibrated questions trigger human validation.
Error tolerance must be set from the downstream decision and failure cost. A threshold chosen after seeing the outputs is descriptive, not confirmatory.
Report failures with the headline result.
- R1Population and leakage
Sampling frame, field dates, demographic coverage, construction data, retrieval sources, and leakage controls.
- R2Configuration and drift
Model and prompt versions, parameters, tools, run dates, repeated-run variance, and material changes between versions.
- R3Distribution and subgroup error
Sample sizes, point estimates, uncertainty, calibration, variance, tails, correlations, missingness, and every prespecified subgroup.
- R4Decision rule and escalation
Preregistered tolerance, whether it was met, sensitivity analyses, excluded uses, and the trigger for fresh human evidence.
Primary sources behind this framework.
The reviewed record contains supportive, mixed, cautionary, and architecture evidence. Findings remain bounded to each study’s model, sample, task, and date.
- R.001Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
Bisbee, Clinton, Dorff, Kenkel & Larson · Political Analysis 32(4) · 2024 · doi:10.1017/pan.2024.5
- R.002Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, Busby, Fulda, Gubler, Rytting & Wingate · Political Analysis 31(3) · 2023 · doi:10.1017/pan.2023.2
- R.003Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
Aher, Arriaga & Kalai · ICML / PMLR 202 · 2023 · pmlr:v202/aher23a
- R.004Towards Measuring the Representation of Subjective Global Opinions in Language Models
Durmus et al. · COLM 2024 · 2024 · openreview:zl16jLb91v
- R.005LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
Park et al. · arXiv:2411.10109, revised June 2026 · 2024 · arxiv:2411.10109
- R.006Generative Agents: Interactive Simulacra of Human Behavior
Park, O’Brien, Cai, Morris, Liang & Bernstein · UIST 2023 · 2023 · doi:10.1145/3586183.3606763
- R.007Can AI Language Models Replace Human Participants?
Dillion, Tandon, Gu & Gray · Trends in Cognitive Sciences 27(7) · 2023 · doi:10.1016/j.tics.2023.04.008