Research observatory / editorial feed v0.1
What does the evidence let us claim?
A living map of synthetic-persona research: original sources, study design, human reference, result, and boundary—kept separate so a promising result cannot become a universal claim.
Editorial synthesis / bounded
Useful as a model. Unsafe as an unquestioned substitute.
Across the reviewed studies, synthetic participants can reproduce some aggregate patterns and become more useful when grounded in real people or records.
The same literature documents compressed variance, subgroup error, prompt and version drift, stereotype effects, and weak individual prediction. Validity is task-specific; fresh human evidence remains the reference for consequential decisions.
SYNTHESIS v0.1 · SOURCE SET R.001–R.007 · NOT A META-ANALYSISPractical protocol
Minimum evaluation record
A study should answer all six checks before its result is used to compare a tool or support a decision.
Name the claim
Separate ideation quality, aggregate distribution fit, subgroup inference, and individual prediction.
The intended decision and failure cost are stated before outputs are inspected.
Hold out humans
Use human responses or behaviour that were not used to construct, retrieve for, or tune the personas.
Reference data, sampling frame, dates, exclusions, and leakage controls are disclosed.
Test distributions
Report variance, calibration, subgroup error, correlations, and rare-response coverage—not mean agreement alone.
The metric matches the downstream decision and includes uncertainty intervals.
Repeat the system
Repeat prompts, runs, dates, and model versions; lock configurations needed for reproduction.
Run-to-run and version drift stay within a preregistered tolerance.
Probe failure modes
Check stereotype amplification, persona collapse, refusals, contamination, missing minorities, and prompt sensitivity.
Failures are reported by subgroup and cannot be averaged away by overall fit.
Escalate to people
Define when synthetic evidence is exploratory and when fresh human research remains mandatory.
High-consequence, novel, heterogeneous, or weakly calibrated questions trigger human validation.
Editorially reviewed / original sources
Evidence feed
Sorted by evidentiary usefulness and balance, not recency, citation count, sponsor, or commercial relationship.
Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
Bisbee, Clinton, Dorff, Kenkel & Larson · Political Analysis 32(4) · 2024 · doi:10.1017/pan.2024.5
- Question in scope
- Population inference from persona-conditioned survey responses
- Human reference
- 2016–2020 American National Election Study
- Reported finding
- Synthetic averages sometimes looked close, but response variance, regression estimates, prompt sensitivity, and results over time did not reliably reproduce the human survey.
- Do not generalize beyond
- One closed model family, US political attitudes, and a defined collection period; it does not establish failure for every architecture or research task.
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, Busby, Fulda, Gubler, Rytting & Wingate · Political Analysis 31(3) · 2023 · doi:10.1017/pan.2023.2
- Question in scope
- Demographically conditioned political-opinion distributions
- Human reference
- Multiple large US survey samples
- Reported finding
- Conditioning GPT-3 on real respondent backstories produced subgroup response patterns the authors describe as algorithmic fidelity.
- Do not generalize beyond
- Evidence is tied to the studied model, US samples, prompts, and outcomes. Distributional resemblance does not imply individual prediction or causal validity.
Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
Aher, Arriaga & Kalai · ICML / PMLR 202 · 2023 · pmlr:v202/aher23a
- Question in scope
- Replication of classic behavioural experiments
- Human reference
- Published human-study effects across four tasks
- Reported finding
- Recent models reproduced three studied effects, while the wisdom-of-crowds task exposed a systematic hyper-accuracy distortion.
- Do not generalize beyond
- Replicating an aggregate effect is not the same as recovering the full human distribution, mechanism, or subgroup structure.
Towards Measuring the Representation of Subjective Global Opinions in Language Models
Durmus et al. · COLM 2024 · 2024 · openreview:zl16jLb91v
- Question in scope
- Cross-national opinion representation and prompting
- Human reference
- GlobalOpinionQA cross-national survey distributions
- Reported finding
- Default model responses aligned more closely with some countries; country prompting shifted alignment but could introduce cultural stereotypes.
- Do not generalize beyond
- The work evaluates whose distributions model responses resemble, not commercial persona systems or individual-level forecasting.
LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
Park et al. · arXiv:2411.10109, revised June 2026 · 2024 · arxiv:2411.10109
- Question in scope
- Interview-grounded individual simulation across outcomes
- Human reference
- 1,052 people, self-retest surveys, traits, and experiments
- Reported finding
- Interview-grounded agents reached 85% of participants’ own two-week retest accuracy on General Social Survey items and reduced some demographic accuracy gaps versus demographic-only agents.
- Do not generalize beyond
- This is a preprint; the reported ratio is relative to human self-retest accuracy and should not be read as 85% absolute accuracy on arbitrary decisions.
Generative Agents: Interactive Simulacra of Human Behavior
Park, O’Brien, Cai, Morris, Liang & Bernstein · UIST 2023 · 2023 · doi:10.1145/3586183.3606763
- Question in scope
- Memory, reflection, planning, and believable social simulation
- Human reference
- Human ratings of agent believability; no held-out survey distribution
- Reported finding
- The memory–reflection–planning architecture produced more believable individual and emergent behaviours than ablated alternatives.
- Do not generalize beyond
- Believability is not population validity. This paper supports an agent architecture, not the replacement of human research participants.
Can AI Language Models Replace Human Participants?
Dillion, Tandon, Gu & Gray · Trends in Cognitive Sciences 27(7) · 2023 · doi:10.1016/j.tics.2023.04.008
- Question in scope
- Conceptual conditions for AI-as-participant research
- Human reference
- Review and illustrative human-alignment evidence
- Reported finding
- The article identifies domains where model judgments can align with people while arguing that interpretability, population coverage, and construct validity prevent general replacement.
- Do not generalize beyond
- This is a short conceptual review, not a general-purpose benchmark or product evaluation.
OpenAlex discovery / daily refresh
New literature watch
Machine-discovered candidates only. Presence here is not inclusion, endorsement, quality review, or support for a finding.
DISCOVERY SERVICE UNAVAILABLE
The editorially reviewed feed remains available. Candidate discovery will retry on the next daily refresh.
Query and filtering rules are published in the site repository. Citation counts are discovery metadata, not quality scores. Metadata can be incomplete; verify every candidate at its primary source.
Feed governance
How a paper enters the reviewed feed
The Minds experiments literature digest was used to find candidates. Editorial statements above were checked against original publisher, proceedings, OpenReview, or arXiv records.
- 01
Discover broadly
Automated search and reader nominations create candidates. Discovery does not confer evidence status.
- 02
Verify the record
Confirm title, authors, venue, identifier, publication status, retraction state, and the presence of an appropriate human reference.
- 03
Extract finding and boundary
Record what the study directly tested and an equally prominent statement of what the result does not establish.
- 04
Disclose conflicts
Minds-affiliated work must be labeled first-party and reviewed under the published recusal. None is included in this initial independent literature set.
- 05
Version corrections
Material changes update the review date and feed version; retractions or major corrections remain visible in the record.