Files and versions

File	Contract	Purpose
leaderboard.json	v12 schema	Index of the `leaderboard/` part files, with row counts and SHA-256 checksums. Releases before v12 carry every row in this one file.
`leaderboard/<leaderboard>.json` and `leaderboard/<leaderboard>/<game>.json`	v12 schema	Canonical track rows for one leaderboard, and its per-game rows for one game. Concatenating the parts in index order gives the full ordered row list.
leaderboard.csv	v12 projection	Flat projection of the same ordered rows in one file.
leaderboard.parquet	v12 metadata	Arrow-typed identity, status, rank, eligibility, score, and lossless row JSON in one Snappy-compressed file.
`publication_readiness.json`	v3 block	Earned evidence states: `preliminary` (every draw at the 48-draw checkpoint is a complete two-seat pairing; no precision claim) and `interval_validated` (a 95% interval from a method validated across programs, not guaranteed for each program; it describes the target-weighted pairing score from 0 to 1, not the 0–100 score; `precision_target_met` says whether its half-width is within 0.05; published only for an interval method the release's state policy qualifies; a state carried forward from an earlier target with another method is published as `preliminary` when its interval came from the preliminary checkpoint, and not at all otherwise). When the state policy qualifies a method, `interval_validation` records its validation: the analysis and its SHA-256 (`qualification_evidence`), the date (`validated_on`), the evidence it ran on (`validation_evidence_commit_sequence`, `validation_evidence_versions`), and `carry_over`: the validation carries over across engine and SDK upgrades, so it applies to results pooled across versions. Every state carries the `population_revision` (SHA-256 of the target population), the `evidence_commit_sequence`, and the `earned_report_sha256` and `earned_state_policy_sha256` it was earned under; a row state also carries its `target_revision` (SHA-256 of the row's whole target, including its own program weights). `counts.target_excluded_programs` is the number of programs that cannot be scored reliably (quarantined by the benchmark worker; a crash or illegal move counts as the faulting program's loss instead) and so leave every target; `target_exclusion_policy` names the list's rule (`readiness_target_exclusions_v2`). `current_state` marks a state carried forward from an earlier target and gives its state under the current target. Programs are keyed by `E-` ID and appear only beside rated rows; model/setting rows by pooled model key, `request_mode` and `effective_request_value`, with per-game states keyed by public game alias. An absent key means no earned state: not yet earned, target undefined, an unrated row, or no single matching public row.
charts_data.json	v2 schema	Overall rank, status, reasons, match, game, cost, and chart series.
benchmark-summary.json	v4 schema	Release-bound population counts and ranked Reasoning Variants leaders.
release-manifest.json	v6 schema	Release identity, counts, hashes, citation, and immutable URLs.
route-inventory.json (prepared pair)	v1 schema	Canonical-to-physical mappings, aliases, hashes, and per-route release evidence.

Row identity and joins

canonical_public_row_identity is the stable public key for one row in one benchmark view. Reasoning Variants identities bind the model to an evaluated-setting tuple; equal-looking bracket names are never a join key. Join JSON, CSV, and Parquet on this field. Join Charts to Overall rows with its same identity.

evaluated_setting_identity is evaluated_setting_identity.v1: canonical provider route and model, request mode and effective request value, prompt and prompt-profile identities, output mode, reasoning policy and catalog identity, plus resolution_status. exact means the complete tuple was retained; opaque_historical preserves older evidence without inventing missing request details.

Ranking, eligibility, and nulls

ranking_status is ranked, provisional, partial, or not_applicable. Overall rows with ranked or partial status share one score-ordered official rank sequence. A partial row keeps eligibility_flag=false and its reason codes because its evidence is incomplete. Reasoning Variants with provisional status remain unranked. A null rank/official_rank means no official rank, not zero or tied-last.

Reason codes are missing_bracket_scores, unknown_capability_evidence, missing_or_non_distinct_settings, untested_or_ineligible_settings, missing_generation_quality, generation_incomplete, below_generation_quality_threshold, fixed_coverage_incomplete, and grouped_stability_incomplete. Current official-qualification rows distinguish an unfinished generation cohort, an actual generation-quality failure after completion, incomplete fixed opponent-and-seat coverage, and grouped rating evidence that is not yet stable. Historical rows without an official-qualification marker retain the generation-only compatibility reason. A Partial 3/3 row has all three score values but fails at least one other eligibility condition named by these codes.

Other null numeric fields mean the measurement is unavailable or not applicable. They never mean zero. Empty arrays mean no observations of that kind were retained.

Measures and units

score and min/max displays use the documented public 0–100 scale. No uncertainty is published or displayed (uncertainty is null).
headline_games on a track row (JSON and Parquet row JSON; headline_games_json in CSV) gives each game's headline value, keyed by public game alias: score (the working programs' scores weighted by the attempts that produced them, plus 0 for each model-caused failed generation attempt, divided by those attempts), playable (the working programs' attempt-weighted mean), attempts and failed_attempts. A row's score is the mean of its games' score. Overall is the exception: its score is the mean of the setting scores, and its game values are each game's mean over the settings, so they need not average to it. Game rows (row_kind game) and the Charts series per_game_tracks and per_game_all are working-program strength: they leave failed attempts out, so they do not reproduce the headline. Charts carries the headline values as headline_games_tracks and headline_games_all.
raw_elo and raw_elo_rating keep their names for compatibility. In this release they hold the B1 rating (a Bradley–Terry fit, points around 1500 with the game's programs centred) described on the Methodology page. They compare entrants within the same game, not with releases rated otherwise. No rating uncertainty is published.
Match counts are integer participant results; for model-setting rows, rated_matches = wins + losses + draws.
Opponents played (an optional leaderboard column, hidden by default) is Σ played ÷ Σ owed over the row's player programs, shown as a whole percentage rounded down. For each program in the Reasoning Variants data of entrants.json, owed_seat_pairings = 2 × (programs in the game − 1): the full matrix owes one rated match against every other program of the game, once with each moving first. played_seat_pairings counts those with at least one completed rated match. Overall rows pool the model's programs from the Baseline, Balanced and Intensive views; other rows pool the programs of their own view. Releases whose data was produced before October 8, 2026 lack both fields; they instead owe each other program once (programs in the game − 1) and count distinct_rated_opponents as played.
Rates are fractions from 0 to 1 in data, even when the site displays percentages.
Token counts are tokens; latency is milliseconds; generation time is seconds; USD fields are US dollars.

Cost evidence

Cost is an equal-weight mean across unique model/game/evaluated-setting cells. evidence_basis, cost_basis_counts, and pricing_table_versions disclose whether values are billed, provider-reported, versioned standard-list estimates, legacy, or combined. Exact zero is 0; a positive sub-cent value remains positive and must not be rounded to a displayed zero.

Example: a row with mean 0.0042, evidence basis “Standard list-price estimate,” and pricing revision pricing-v3 means an estimated 0.42 cents per unique cell under that revision—not a provider invoice.

Population counts

model_family_count counts unique model families. Model-setting counts separate total, ranked, and provisional reasoning variants. ranked_overall_model_count keeps its compatibility meaning of complete-evidence Overall rows; the total number of official Overall ranks is ranked_overall_model_count + partial_overall_model_count. unique_code_entrant_count deduplicates public code identities across the suite; per_game_unique_entrant_counts gives the corresponding count for each public game alias. model_count and reasoning_variant_count are temporary compatibility aliases.

Data dictionary