Protocol / v0.2 draft
Evidence before rank.
The Index separates discoverability from validation and reports each evidentiary property independently.
Six checks, not one maturity score.
A public artifact does not imply human validity. A human comparison does not imply independent reproduction. Each check stands on its own.
Claim documented
An attributable source establishes that a specific claim exists; it does not establish performance.
Artifact available
Code, data, prompts, or an executable artifact are persistently accessible for inspection.
Method reviewed
The disclosed design and analysis have been screened against a published protocol.
Independently reproduced
A team independent of the vendor obtained materially consistent results from disclosed artifacts.
Human-reference validation
Outputs were compared with an appropriate human sample or real-world outcome under a disclosed design.
Limitations disclosed
Known scope limits, missing evidence, conflicts, and non-generalizable conditions are explicit.
Six operations, with an audit trail.
- 01
Scope
Determine whether the record is a research platform, respondent system, audience model, copilot, benchmark, or adjacent service.
- 02
Source
Capture canonical first-party documentation and seek independent validation, reference data, or reproducible code.
- 03
Atomize
Split broad marketing language into claims that can be supported, contradicted, or marked unknown individually.
- 04
Classify
Record each evidence check independently for the claim set, never as a company-wide credibility score.
- 05
Challenge
Give the organization a correction window and record unresolved disagreements without allowing paid editorial control.
- 06
Publish
Show sources, access dates, limitations, conflicts, and the protocol version beside the result.
Freeze the test before seeing the answer.
Every comparative benchmark must publish its hypotheses, reference sample, exclusions, prompts, tool versions, primary measures, subgroup analyses, stopping rule, and statistical plan before evaluation begins.
The first Index benchmark is a draft and has not been preregistered or run. No leaderboard result is implied.
Publish uncertainty before aggregation.
Results must expose sample size, effect or error estimate, uncertainty interval, repeated-run variance, subgroup error, missingness, model and prompt version, and the human reference. Composite scores require preregistered weights.
Not every task should collapse into one score.
Exploratory ideation, qualitative thematic overlap, distribution matching, behavioural prediction, and data augmentation require different references and error tolerances. The Index will only rank systems within a defined task and protocol version.
Fluent output, persona detail, and respondent-like language are not treated as validation by themselves.