Synthetic Research IndexSubmit a tool

Benchmark / public protocol

Measure the decision, not the performance.

A public evaluation framework for where synthetic systems agree with human references, where they compress variance, and where plausible language hides decision error.

Pre-result status

No comparative finding has been produced.

The candidate protocol remains open to critique. Tool selection, primary outcomes, reference data, exclusions, subgroup tests, and statistical analysis must be frozen in a timestamped public registration before the first scored run.

Inspect candidate preregistration OSRB-001 →

DimensionPrimary measureReferenceFailure signal
FidelityDistribution and theme agreementHeld-out human sampleDirection or relationship flips
ReliabilityBetween-run varianceRepeated identical runsUnstable conclusion
CalibrationConfidence vs. observed errorTask-specific ground truthConfident but wrong
TraceabilitySource and configuration coverageAudit checklistUnrecoverable provenance
Decision relevanceRecommendation agreementPredefined decision ruleSame evidence, different action
B.01

Reference sets

Use datasets whose sampling, questionnaire, field dates, weighting, and limitations are known. Hold back a portion from system builders where leakage would invalidate the task.

B.02

Blind runs

Normalize briefs and outputs before review. Record models, prompts, grounding sources, run dates, costs, and every post-processing step.

B.03

Error analysis

Publish subgroup error, variance compression, refusals, unsupported claims, and relationship reversals—not only an aggregate score.

B.04

Decision tests

Translate outputs into predefined decisions. Measure whether systems merely sound similar or actually support the same action.

B.05 / REQUIRED REPORT

A number is incomplete without its uncertainty.

estimate + unit95% intervalhuman nsynthetic runssubgroup errormissingnessmodel + prompt versiondecision threshold

Aggregate rankings remain prohibited unless the weighting rule and failure costs are preregistered.

B.06 / REGISTRATION FIELDS

What must be frozen first.

  1. Research questions and confirmatory hypotheses
  2. Reference population, sampling frame, field dates, weighting, and exclusions
  3. Tool inclusion rule, model versions, prompts, grounding, and post-processing
  4. Primary and secondary outcomes, subgroup analyses, and multiplicity handling
  5. Missing-data treatment, stopping rule, decision thresholds, and release plan

Call for collaborators

Contribute a human reference dataset or evaluation protocol.

We are looking for publishable or safely aggregated paired studies, research-method expertise, and an external reviewer who passes the published conflict standard.

Reviewer call