Files and versions
| File | Contract | Purpose |
|---|---|---|
| leaderboard.json | v11 schema | Canonical track and per-game rows. |
| leaderboard.csv | v11 projection | Flat projection of the same ordered rows. |
| leaderboard.parquet | v11 metadata | Arrow-typed identity, status, rank, eligibility, score, and lossless row JSON. |
| charts_data.json | v2 schema | Overall rank, status, reasons, match, game, cost, and chart series. |
| benchmark-summary.json | v4 schema | Release-bound population counts and ranked Reasoning Variants leaders. |
| release-manifest.json | v6 schema | Release identity, counts, hashes, citation, and immutable URLs. |
| route-inventory.json (prepared pair) | v1 schema | Canonical-to-physical mappings, aliases, hashes, and per-route release evidence. |
Row identity and joins
canonical_public_row_identity is the stable public key for one row in one benchmark view. Reasoning Variants identities bind the model to an evaluated-setting tuple; equal-looking bracket names are never a join key. Join JSON, CSV, and Parquet on this field. Join Charts to Overall rows with its same identity.
evaluated_setting_identity is evaluated_setting_identity.v1: canonical provider route and model, request mode and effective request value, prompt and prompt-profile identities, output mode, reasoning policy and catalog identity, plus resolution_status. exact means the complete tuple was retained; opaque_historical preserves older evidence without inventing missing request details.
Ranking, eligibility, and nulls
ranking_status is ranked, provisional, partial, or not_applicable. Overall rows with ranked or partial status share one score-ordered official rank sequence. A partial row keeps eligibility_flag=false and its reason codes because its evidence is incomplete. Reasoning Variants with provisional status remain unranked. A null rank/official_rank means no official rank, not zero or tied-last.
Reason codes are missing_bracket_scores, unknown_capability_evidence, missing_or_non_distinct_settings, untested_or_ineligible_settings, missing_generation_quality, and below_generation_quality_threshold. A Partial 3/3 row has all three score values but fails at least one other eligibility condition named by these codes.
Other null numeric fields mean the measurement is unavailable or not applicable. They never mean zero. Empty arrays mean no observations of that kind were retained.
Measures and units
score, min/max, and uncertainty displays use the documented public 0–100 scale.- Raw Elo fields are Elo-like points and compare entrants within the same game and evidence scope.
- Match counts are integer participant results; for model-setting rows,
rated_matches = wins + losses + draws. - Rates are fractions from 0 to 1 in data, even when the site displays percentages.
- Token counts are tokens; latency is milliseconds; generation time is seconds; USD fields are US dollars.
Cost evidence
Cost is an equal-weight mean across unique model/game/evaluated-setting cells. evidence_basis, cost_basis_counts, and pricing_table_versions disclose whether values are billed, provider-reported, versioned standard-list estimates, legacy, or combined. Exact zero is 0; a positive sub-cent value remains positive and must not be rounded to a displayed zero.
Example: a row with mean 0.0042, evidence basis “Standard list-price estimate,” and pricing revision pricing-v3 means an estimated 0.42 cents per unique cell under that revision—not a provider invoice.
Population counts
model_family_count counts unique model families. Model-setting counts separate total, ranked, and provisional reasoning variants. ranked_overall_model_count keeps its compatibility meaning of complete-evidence Overall rows; the total number of official Overall ranks is ranked_overall_model_count + partial_overall_model_count. unique_code_entrant_count deduplicates public code identities across the suite; per_game_unique_entrant_counts gives the corresponding count for each public game alias. model_count and reasoning_variant_count are temporary compatibility aliases.