SRISynthetic Research IndexSubmit a tool
← benchmark

Candidate preregistration / public protocol

Survey-distribution fidelity.

A confirmatory design for measuring agreement with held-out human response distributions—not fluency, plausibility, or marketing claims.

NOT PREREGISTERED YET

This is a candidate record, not a completed external registration. It remains editable until every lock condition is met and a timestamped immutable version is published before scoring.

Research question

One narrow comparison first.

For eligible synthetic-research systems, how closely do closed-ended response distributions reproduce a held-out human reference under a fixed brief and locked configuration?

Eligibility

Capability, not company identity

Systems must accept the same structured audience and questionnaire brief and return respondent-level or aggregate closed-ended distributions. Eligibility is capability-based, not vendor-based.

Confirmatory outcome

Mean Jensen–Shannon distance between synthetic and held-out human response distributions across preregistered items.

Secondary outcomes

  1. S-01Percentage-point mean absolute error
  2. S-02Rank agreement across response options
  3. S-03Between-run variance across at least 10 identical synthetic runs
  4. S-04Maximum preregistered subgroup error

Protocol block D

Design and blinding

  1. D-01

    Use a human reference with documented sampling frame, field dates, questionnaire, exclusions, and weighting.

  2. D-02

    Keep primary human outcomes hidden from system operators until configurations are locked.

  3. D-03

    Set synthetic respondent count equal to the analyzed human count for each task and run each system at least 10 times.

  4. D-04

    Freeze system version, prompts, grounding sources, post-processing, exclusions, and stopping rules before scoring.

  5. D-05

    Blind system identity in analyst-facing output where operationally possible.

Protocol block A

Analysis plan

  1. A-01

    Report point estimates with 95% bootstrap intervals.

  2. A-02

    Apply Holm correction to confirmatory pairwise comparisons.

  3. A-03

    Report missingness, refusals, relationship reversals, and subgroup failures.

  4. A-04

    Keep qualitative theme analysis exploratory and separate from the confirmatory primary outcome.

Release rule

Negative findings stay visible.

Publish all eligible completed runs, including null and adverse findings. No overall ranking may be derived unless its weighting and failure costs are locked in a later version before results are seen.

Protocol block L

Conditions required to lock v1.0

  1. L-01

    An external reviewer has consented, passed a disclosed conflict review, and approved the protocol.

  2. L-02

    Reference dataset, hypotheses, outcomes, exclusions, system configurations, and analysis code are frozen.

  3. L-03

    A timestamped immutable copy or public registry entry is published before the first scored run.

MINDS RECUSAL

Minds is recused from approving the protocol lock, evaluating its own system, resolving its disputes, and authoring the final Minds conclusion. Inspect the enforceable record →

Open methods review

Critique before measurement.

Submit a methodological objection, proposed reference dataset, or reviewer application. Material changes create a new version.

Submit evidenceReviewer call