Files and versions

leaderboard.json (v12 schema): Index of the leaderboard/ part files, with row counts and SHA-256 checksums. Releases before v12 carry every row in this one file.
leaderboard/<leaderboard>.json and leaderboard/<leaderboard>/<game>.json (v12 schema): Canonical track rows for one leaderboard, and its per-game rows for one game. Concatenating the parts in index order gives the full ordered row list.
leaderboard.csv (v12 projection): Flat projection of the same ordered rows in one file. Numbers are rounded to six decimal places.
leaderboard.spreadsheet.csv (Spreadsheet view): The same rows for opening in a spreadsheet. Cells beginning with formula markers or tab/newline controls receive an apostrophe so they import as text, including negative numbers. Use JSON or Parquet for full precision, or the canonical CSV above (six decimal places).
leaderboard.parquet (v12 metadata): Arrow-typed identity, status, rank, eligibility, score, and lossless row JSON in one Snappy-compressed file.
publication_readiness.json (v3 block): Earned evidence states: preliminary (every draw at the 48-draw checkpoint is a complete two-seat pairing; no precision claim) and interval_validated (a 95% interval from a method validated across programs, not guaranteed for each program; it describes the target-weighted pairing score from 0 to 1, not the 0–100 score; precision_target_met says whether its half-width is within 0.05; published only for an interval method the release's state policy qualifies; a state carried forward from an earlier target with another method is published as preliminary when its interval came from the preliminary checkpoint, and not at all otherwise). When the state policy qualifies a method, interval_validation records its validation: the analysis and its SHA-256 (qualification_evidence), the date (validated_on), the evidence it ran on (validation_evidence_commit_sequence, validation_evidence_versions), and carry_over: the validation carries over across engine and SDK upgrades, so it applies to results pooled across versions. Every state carries the population_revision (SHA-256 of the target population), the evidence_commit_sequence, and the earned_report_sha256 and earned_state_policy_sha256 it was earned under; a row state also carries its target_revision (SHA-256 of the row's whole target, including its own program weights). counts.target_excluded_programs is the number of programs that cannot be scored reliably (quarantined by the benchmark worker; a crash or illegal move counts as the faulting program's loss instead) and so leave every target; target_exclusion_policy names the list's rule (readiness_target_exclusions_v2). current_state marks a state carried forward from an earlier target and gives its state under the current target. Programs are keyed by E- ID and appear only beside rated rows; model/setting rows by pooled model key, request_mode and effective_request_value, with per-game states keyed by public game alias. An absent key means no earned state: not yet earned, target undefined, an unrated row, or no single matching public row.
charts_data.json (v2 schema): Overall rank, status, reasons, match, game, cost, and chart series.
benchmark-summary.json (v4 schema): Release-bound population counts and ranked Reasoning Variants leaders.
release-manifest.json (v6 schema): Release identity, counts, hashes, citation, and immutable URLs.
route-inventory.json, in the prepared release pair (v1 schema): Canonical-to-physical mappings, aliases, hashes, and per-route release evidence.

Row identity and joins

canonical_public_row_identity is the public key for one row in one benchmark view of this release. Reasoning Variants identities bind the model to an evaluated-setting tuple; equal-looking setting labels (Baseline, Balanced, Intensive) are never a join key. Join JSON, CSV, and Parquet on this field. Join Charts to Overall rows with its same identity. Across releases, reference a result by public_release_id together with this key (see Importing results).

evaluated_setting_identity is evaluated_setting_identity.v1: canonical provider route and model, request mode and effective request value, prompt and prompt-profile identities, output mode, reasoning policy and catalog identity, plus resolution_status; on Reasoning Variants rows, also thinking_budget_tokens when the request set a manual thinking budget (see Reasoning settings). exact means the complete tuple was retained; opaque_historical preserves older evidence without inventing missing request details.

Reasoning settings

On Reasoning Variants rows (track full_freedom/mixed, “Model + reasoning setting” on the site), the provider request is recorded in evaluated_setting_identity: request_mode and effective_request_value. Classify a row from those two fields, never from its label.

resolution_status describes the whole identity: route, model, request, prompt, output mode, policy and catalog. opaque_historical means at least one of them was not retained, often the prompt or output identity. It does not mean the effort is unknown.
thinking_budget_tokens is the manual extended-thinking budget an effort request sent together with its level. Only Claude Opus 4.5 sends one: 16,000 tokens with Medium and 32,000 with High. It appears on Reasoning Variants rows and on each program in entrants.json. Baseline, Balanced and Intensive rows leave it out because they can pool several requests: Claude Opus 4.5's Balanced and Intensive rows also hold programs that sent the older shared budget. On other effort rows the field is absent because no budget was sent. Rows of the budget class sent one historical shared budget whose size is not recorded. Releases published before the field existed leave it out, and there the Opus 4.5 budget shows only in the label. canonical_public_row_identity does not include the field, since the route, model and effort already fix the budget.
reasoning_setting is the release's public setting key, such as historical-provider-setting-intensive, with -variant-N when a model has several requests in one group; variant numbers follow position within the release. reasoning_provenance.requested_reasoning_effort, provider_reasoning_params.requested_effort and the CSV column requested_reasoning_effort hold the same key. None of them is the literal request. In entrants.json, reasoning_setting is each program's recorded value.
On these rows, evidence_status and bracket_eligible in provider_reasoning_params follow resolution_status: while no row's identity is exact, they read retained_literal_legacy and false on every row and say nothing about whether the request is recorded. On the Baseline, Balanced and Intensive leaderboards they tell whether a group's setting is the model's documented one.
reasoning_level is the display label: the current capability catalog's wording for the recorded value, so its text can change between releases. Releases up to gb2-20261107-a12a38c1543d label a model without reasoning controls "Provider default (reasoning controls unavailable)", as in the import examples below; later releases label it "Reasoning controls unavailable". reasoning_bracket is the group, and is null for a shared budget. Neither is a join key.
The Baseline, Balanced and Intensive leaderboards have one row per model and group. A row there can pool several requests of its group, so it is not one setting; use Reasoning Variants rows for single settings.

Request classes, by request_mode and effective_request_value:

literal_effort: effort with a level such as low or xhigh. A recorded effort level. thinking_budget_tokens, when present, is a thinking budget sent with it. 199 rows in this release, 189 ranked.
disabled: none under effort or toggle. Reasoning turned off. 57 rows in this release, 47 ranked.
enabled_without_levels: toggle with enabled. Reasoning turned on; the model has no levels to choose. 20 rows in this release, 16 ranked.
field_omitted: omitted with none. No reasoning field was sent. Not the same as disabled. 4 rows in this release, 3 ranked.
provider_default: omitted with default. The model offers no reasoning control; the provider default applied. 3 rows in this release, 2 ranked.
shared_budget: budget. One historical thinking budget that served more than one reasoning group; the row has no group. 1 row in this release, 1 ranked.
other_recorded_value: any other recorded value. A recorded request outside the classes above. 0 rows in this release, 0 ranked.
unresolved: an effective_request_value of historical-… or unknown_legacy. No row-level request value. Unknown is not zero, disabled, low or the provider default. 0 rows in this release, 0 ranked.

Importing results

Pin one release. Use public_release_id from release-manifest.json and cite the immutable releases/<public_release_id>/ copy. Check leaderboard.json against the manifest's artifact_sha256, and each part's size and SHA-256 against the index. On rows, results_cut_id equals public_release_id; snapshot_id, comparability_tuples[].results_cut_id and the results_cut_id of release-registry.json and release-lineage.json entries carry the internal cut identifier; cite the release ID instead.
Download with your own User-Agent. The DuelLab sites refuse Python's default user agent (Python-urllib/3.x, which pandas.read_csv also sends when it reads a URL) with HTTP 403 and the body error code: 1010. Send your own, for example pd.read_csv(url, storage_options={"User-Agent": "MyDataClient/1.0"}) or urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "MyDataClient/1.0"})). Browsers, curl and the requests library are not refused. The Hugging Face mirror serves the files of a recent release without this restriction; check its public_release_id.
Check that the release is registered. release-registry.json lists every release that has passed our live verification, with the SHA-256 of its immutable releases/<public_release_id>/release-manifest.json. We add each release to it after that verification, as part of publishing the release. The release-lineage.json inside a package lists the package itself as current_unregistered with a null manifest_sha256, because a file in the package cannot carry the hash of the manifest that hashes it. A new release appears there only after its live verification; if you import only registered releases, wait for its entry. To check a release yourself, compare the SHA-256 of its immutable release-manifest.json with the registry entry, then compare the SHA-256 of each file you import with the manifest's artifact_sha256 entry for benchmark/releases/<public_release_id>/<file>.
Choose the view. overall has one row per model. A row with an official rank scores the mean of the model's three group scores; a model with results for fewer groups is listed without a rank, and its score averages only the groups it has. full_freedom/mixed (Reasoning Variants) has one row per evaluated request; use it for single settings. The Baseline, Balanced and Intensive leaderboards have one row per model and group, which can pool several requests.
Select eligible rows. For fully eligible results, keep ranking_status ranked with eligibility_flag true. A rank alone is not enough: partial Overall rows carry official ranks with incomplete evidence. After this filter the source ranks have gaps; keep them, or label positions you number yourself as yours. Provisional rows have no rank, and their scores rest on incomplete evidence; never sort them among ranked rows.
Keep Overall, single settings and best settings apart. A ranked Overall score is DuelLab's mean over a model's three groups. A “best setting” is your own statistic over Reasoning Variants rows: say which rows it chose from and how it breaks ties, keep the chosen row's key, and do not call it Overall.
Map settings by class. Use the classes under Reasoning settings. Do not map an unresolved value to a standard effort name, and do not treat unknown as zero, disabled, low or the provider default.
Keep the source identity. model_slug identifies the model in a release; dated versions have their own slugs (a -0902 suffix, for example) and separate rows and scores. model_display_name is for display and need not be unique. On Reasoning Variants rows, evaluated_setting_identity.provider_model is the provider's model identifier. Store these beside your own catalog ID.
Reference a result by release and key. canonical_public_row_identity identifies a row within one release. Reference an imported result as public_release_id plus that key. A key can change between releases without any change in the result, since variant numbers are positional. Overall, Baseline, Balanced and Intensive keys change only with the row's evidence or that model's own reasoning settings (the provider model its catalog entry names, the route, the settings or whether reasoning can be turned off), never with another model's catalog entry. Releases up to 3 November 2026 hashed the whole reasoning catalog into every Overall key and into some group keys, so those changed whenever any model's entry did, and the first release after that changed them once. Establish any cross-release match explicitly.
Keep the precision. JSON and Parquet carry full double precision; the CSV is rounded to six decimal places. Consecutive releases can differ by less than a displayed digit, so a rounded value does not identify its release on its own. Compare at half a displayed unit.
Attribute. The data is published under the manifest's data_license (data_license_url), with its citation (citation_url). Link the immutable release and state your selection rule.

Examples

Rows of release gb2-20261103-eaea31df94e6, with their fields copied from its immutable files, so their ranks and statuses are that release's and can differ from the current data. release-registry.json lists it with manifest SHA-256 5627c77691e8d2fd0fb092240cb105d71fbba88e93798341aa341a803c905bf2. Each row key links to the part file that holds the whole row.

GPT-6 Astra (Reasoning Variants), reasoning_level XHigh; request effort / xhigh; ranked, rank 3; class literal_effort. A recorded effort level. The identity is opaque_historical only because the prompt and output identity were not retained; the effort is known. Row: publication-row.v1:mixed:gpt-6-astra:af89b97f9d676a8c
Pareto 26.10 (Reasoning Variants), reasoning_level Provider default (reasoning controls unavailable); request omitted / default; ranked, rank 7; class provider_default. The model offers no reasoning control, so the provider default applied. It is not disabled and not a low effort. Row: publication-row.v1:mixed:unbiased-pareto-26-10:cede2d03a653c56c
Glimpse (Reasoning Variants), reasoning_level Reasoning enabled; request toggle / enabled; ranked, rank 57; class enabled_without_levels. Reasoning switched on by a toggle with no levels; there is no effort value to map. Row: publication-row.v1:mixed:elorian-glimpse:d613dad94964ec86
Claude Opus 4.5 (Reasoning Variants), reasoning_level Medium effort + 16,000 thinking tokens; request effort / medium, thinking_budget_tokens 16000; ranked, rank 107; class literal_effort. Recorded effort medium, sent with a manual 16,000-token thinking budget, which thinking_budget_tokens records; the High row sent 32,000. The budget is not part of the row key. Row: publication-row.v1:mixed:claude-opus-4-5:e2ff6744eb1bb7c6
Claude Opus 4.5 (Reasoning Variants), reasoning_level Thinking field omitted; request omitted / none; ranked, rank 124; class field_omitted. No reasoning field was sent. Not the same as disabled. Row: publication-row.v1:mixed:claude-opus-4-5:959db58a0f4d59b9
GLM-5.2 (Reasoning Variants), reasoning_level Reasoning disabled (historical request variant 1); request effort / none; provisional, not eligible, listing_status duplicate_variant_not_listed; class disabled. Reasoning off through the effort parameter. A recorded owner decision lists only the model's other reasoning-off row (variant 2, below), so this row keeps its score but is Provisional, with reason other_request_variant_listed and listing_status duplicate_variant_not_listed, and the pages do not show it. Variant numbers are positional within a release. Row: publication-row.v1:mixed:z-ai-glm-5-2:5ea14bc9644b6a2b
GLM-5.2 (Reasoning Variants), reasoning_level Reasoning disabled (historical request variant 2); request toggle / none; ranked, rank 184; class disabled. Also reasoning off, but through a toggle: a different request with its own row and score, and the row the pages list. Keyed by model and effort alone, these two rows would collide. Row: publication-row.v1:mixed:z-ai-glm-5-2:184634950256ca6a
MiMo-V2.5 (Reasoning Variants), reasoning_level High; request effort / high; provisional, not eligible, listing_status cannot_be_ranked; class literal_effort. Provisional, and every missing game failed for a cause outside the model that a new request cannot repeat, so listing_status is cannot_be_ranked: the pages mark it Can't be ranked in the Not ranked list. Its score rests on incomplete evidence. Row: publication-row.v1:mixed:xiaomi-mimo-v2-5:024b37c390436d5a
Claude Opus 4.5 (Reasoning Variants), reasoning_level Historical thinking-budget setting; request budget / legacy_shared_thinking_budget; ranked, rank 138; class shared_budget. One historical thinking budget served more than one reasoning group, so the row has no group (reasoning_bracket is null). Row: publication-row.v1:mixed:claude-opus-4-5:1f51e395856c1899
Claude Haiku 5.5 (Overall), reasoning_level Overall Models; request —; partial, rank 23, not eligible. Partial: an official Overall rank with incomplete evidence (eligibility_flag false). Selecting ranking_status=ranked and eligibility_flag=true leaves it out, and the rows after it keep their source ranks. Row: publication-row.v1:overall:claude-haiku-5-5:c169a5f28df68275
Pareto 26.10 (Overall), reasoning_level Overall Models; request —; provisional, not eligible. Provisional: no rank. Its score averages fewer than three settings, so it is not comparable with ranked rows and must not be sorted among them. Row: publication-row.v1:overall:unbiased-pareto-26-10:3f4dae3fc99a9dcb
Qwen3.8 Max 0902 (Overall), reasoning_level Overall Models; request —; ranked, rank 26. A dated model version. Its slug and provider model differ from the undated Qwen3.8 Max below; both are separate results. Row: publication-row.v1:overall:qwen-qwen3-8-max-0902:27f0ad746deb6345
Qwen3.8 Max (Overall), reasoning_level Overall Models; request —; ranked, rank 38. The undated version, with its own score. A shared catalogue name is not a join key. Row: publication-row.v1:overall:qwen-qwen3-8-max:5f41df3bb03edf57
DeepSeek V4 Pro 0813 (Overall), reasoning_level Overall Models; request —; ranked, rank 48. A dated model version, separate from the undated DeepSeek V4 Pro below. Row: publication-row.v1:overall:deepseek-deepseek-v4-pro-0813:84c92d6f9c4c6c41
DeepSeek V4 Pro (Overall), reasoning_level Overall Models; request —; ranked, rank 49. The undated version, with its own score. Row: publication-row.v1:overall:deepseek-deepseek-v4-pro:201663eb3d6fa183

Ranking, eligibility, and nulls

ranking_status is ranked, provisional, partial, or not_applicable. Overall rows with ranked or partial status share one score-ordered official rank sequence. A partial row keeps eligibility_flag=false and its reason codes because its evidence is incomplete. Reasoning Variants rows with provisional status remain unranked. A null rank/official_rank means no official rank, not zero or tied-last.

Reason codes are missing_bracket_scores, unknown_capability_evidence, missing_or_non_distinct_settings, untested_or_ineligible_settings, missing_generation_quality, generation_incomplete, below_generation_quality_threshold, fixed_coverage_incomplete, grouped_stability_incomplete, and other_request_variant_listed (a Reasoning Variants row a recorded owner decision does not list; see listing_status). Current official-qualification rows distinguish an unfinished generation cohort, an actual generation-quality failure after completion, incomplete fixed opponent-and-seat coverage, and grouped rating evidence that is not yet stable. Historical rows without an official-qualification marker retain the generation-only compatibility reason. A Partial 3/3 row has all three score values but fails at least one other eligibility condition named by these codes.

Other null numeric fields mean the measurement is unavailable or not applicable. They never mean zero. Empty arrays mean no observations of that kind were retained.

Measures and units

score and min/max displays use the documented public 0–100 scale. No uncertainty is published or displayed (uncertainty is null).
headline_games on a track row (JSON and Parquet row JSON; headline_games_json in CSV) gives each game's headline value, keyed by public game alias: score (the working programs' scores weighted by the attempts that produced them, plus 0 for each model-caused failed generation attempt, divided by those attempts), playable (the working programs' attempt-weighted mean), attempts and failed_attempts. A row's score is the mean of its games' score. Overall is the exception: its score is the mean of the setting scores, and its game values are each game's mean over the settings, so they need not average to it. Game rows (row_kind game) and the Charts series per_game_tracks and per_game_all are working-program strength: they leave failed attempts out, so they do not reproduce the headline. Charts carries the headline values as headline_games_tracks and headline_games_all; its per-game charts read their score (headline score) or playable (working-program strength). Since 24 October 2026 a setting's game rows and per_game_tracks equal its playable values (they pool every working program of the row, weighted by attempts); per_game_all is the mean over the three settings and has a value only where all three do. Charts model rows also carry diagnostics: working_program_strength (a setting's value equals that leaderboard row's conditional_success_score; all is the mean of the settings), program_outcomes (generation outcomes and success_rate, the row's plugin_success_rate), official_rank_by_reasoning and economics_best_value.
Game rows with several programs (from 25 October 2026): a game row (row_kind game) stands for all of a model's programs in that game and setting. score is their working-program strength, weighted by attempts; raw_elo is the attempt-weighted mean of the working programs' raw ratings, weighted like working-program strength (a failed attempt has no program and no rating); wins, losses, draws and rated_matches are the programs' combined record, which measures evidence volume, not strength; contributing_entrants counts the programs, and a track row's entry_count (the Players column) counts the programs of its game rows (Overall: of its settings). Added fields: working_attempts and counted_attempts (“N working of M attempts”: counted attempts are the working ones plus those that failed in model-written code; too-slow programs and provider, API or test-machine faults are not counted) and program_raw_elo_low and program_raw_elo_high (the lowest and highest program rating, only for rows with two or more rated programs; a description of the programs, not a confidence interval). CSV carries the four after headline_games_json; they are absent on track rows and earlier releases. On the Intensive, Balanced and Baseline leaderboards a model row's wins, draws and losses are the sum of its game rows' records (Reasoning Variants rows already were), and Charts' per-game match totals equal the game rows'. Charts model rows carry game_gaps: per view and game without a working-program strength, why it has none: failed (every counted attempt failed, a recorded 0 in the headline), too_slow (rejected by the speed check, not counted), infra_excluded (lost to a provider, API or test-machine fault, not counted), unrated (a working program not rated yet) or untested (no attempt recorded).
listing_status marks Reasoning Variants rows that the pages list apart. cannot_be_ranked: marked Can't be ranked in the Not ranked list, because every missing game failed for a cause outside the model that cannot be repeated (the provider route no longer exists, or a new request would not reproduce the evaluated identity). superseded_variant_not_listed: not listed on the pages, because it is a Provisional request variant that misses only games while a ranked row of the same model, reasoning bracket and request value exists; its programs still count where they counted, including the model's Overall score. duplicate_variant_not_listed: not listed on the pages, because a recorded owner decision lists another complete request variant of the same setting instead; it keeps its score but is provisional with the reason code other_request_variant_listed, so the listed rows' ranks have no gap and the release's ranked counts leave it out. Other rows have no such field. CSV adds a listing_status column at the end only when some row has a value. A model withdrawn from publication by a recorded owner decision (its licence terms) has no rows, programs or match totals in any page or download; its programs still count as opponents, so every score stays as computed (the ranks below its rows close up), and reference_field_manifest.json lists them under the model withdrawn.
conditional_success_score on a track row keeps its field name for compatibility. It is working-program strength, the leaderboard's column of that name (called Playable before 24 October 2026): the mean score of the working programs across games, with failed generation attempts left out. plugin_success_rate is the program success rate (working programs over counted attempts).
raw_elo and raw_elo_rating keep their names for compatibility. In this release they hold the B1 rating (a Bradley–Terry fit, points around 1500 with the game's programs centred) described on the Methodology page. They compare player programs within the same game, not with releases rated otherwise. No rating uncertainty is published.
Match counts are integer participant results; for model-setting rows, rated_matches = wins + losses + draws.
Opponents played (an optional leaderboard column, hidden by default) is Σ played ÷ Σ owed over the row's player programs, shown as a whole percentage rounded down. For each program in the Reasoning Variants data of entrants.json, owed_seat_pairings = 2 × (programs in the game − 1): the full matrix owes one rated match against every other program of the game, once with each moving first. played_seat_pairings counts those with at least one completed rated match. Overall rows pool the model's programs from the Baseline, Balanced and Intensive views; other rows pool the programs of their own view. Releases whose data was produced before October 8, 2026 lack both fields; they instead owe each other program once (programs in the game − 1) and count distinct_rated_opponents as played.
Rates are fractions from 0 to 1 in data, even when the site displays percentages.
Token counts are tokens; latency is milliseconds; generation time is seconds; USD fields are US dollars.

Cost evidence

Cost is an equal-weight mean across unique model/game/evaluated-setting cells. evidence_basis, cost_basis_counts, and pricing_table_versions disclose whether values are billed, provider-reported, versioned standard-list estimates, legacy, or combined. Exact zero is 0; a positive sub-cent value remains positive and must not be rounded to a displayed zero.

Example: a row with mean 0.0042, evidence basis “Standard list-price estimate,” and pricing revision pricing-v3 means an estimated 0.42 cents per unique cell under that revision—not a provider invoice.

Population counts

model_family_count counts unique model families. Model-setting counts separate total, ranked, and provisional reasoning settings, and count every Reasoning Variants row of the downloads once, including rows the pages list apart or leave out (listing_status): the total is the ranked count plus the provisional count. ranked_overall_model_count keeps its compatibility meaning of complete-evidence Overall rows; the total number of official Overall ranks is ranked_overall_model_count + partial_overall_model_count. unique_code_entrant_count deduplicates public code identities across the suite; per_game_unique_entrant_counts gives the corresponding count for each public game alias. model_count and reasoning_variant_count are temporary compatibility aliases.

Field reference

Every column of leaderboard.csv, in file order; the JSON parts and Parquet row JSON use the same names. The last six columns appear only when some row has a value. Rates are fractions from 0 to 1, times are UTC, and an empty or null value means not available or not applicable, never zero.

snapshot_id: Internal cut identifier (legacy name), for example 20261103_094507. Not a citation key.
results_cut_id: On leaderboard rows, and at the top of leaderboard.json, entrants.json and charts_data.json, the public release ID (equal to public_release_id). In comparability_tuples, release-registry.json and release-lineage.json the same name holds the internal cut identifier, the value of snapshot_id. Cite the release ID.
suite_line_id: Benchmark line, gamebench2.
suite_version: Version of the game set and rules, for example 2-known8-r3.
scoring_policy_version: Scoring policy the row was scored under (the manifest's scoring_policy_id).
rating_method: Rating method behind raw_elo and score; bradley_terry_newton is B1.
row_kind: track: a leaderboard row. game: that row's result in one game.
track_id: The leaderboard: overall, full_freedom/mixed (Reasoning Variants), or full_freedom/baseline, full_freedom/balanced and full_freedom/intensive.
track_name: The leaderboard's display name.
game_alias: Public game alias (GB2-01 to GB2-08) on game rows; empty on track rows. Game names are in charts_data.json games.
canonical_public_row_identity: The row's public key within this release (see Row identity and joins).
evaluated_setting_identity_json: evaluated_setting_identity as compact JSON (see Reasoning settings).
eligibility_flag: true only for complete-evidence ranked rows.
rank: Official rank; empty when the row has none, which is not zero or tied-last. The Parquet column is official_rank.
ranking_status: ranked, partial, provisional or not_applicable (game rows); see Ranking, eligibility, and nulls.
ineligibility_reason_codes: Reason codes, separated by | in CSV (an array in JSON); empty for a fully eligible row.
model_slug: The model's identity within the release. Dated versions have their own slugs and rows.
model_display_name: Display name; not unique and not a join key.
provider: The provider company's display name.
provider_key: Stable lower-case provider key.
reasoning_level: Display label of the setting (Overall Models on Overall rows). Its wording can change between releases.
reasoning_bracket: Reasoning group: baseline, balanced or intensive. Empty on Overall rows and for a shared budget.
reasoning_setting: Public setting key (see Reasoning settings). Empty on Overall rows, and on group rows whose setting was not recorded (their reasoning_level says legacy evidence incomplete).
measured_at: Game rows: UTC time of the latest recorded match of the row's programs in that game. Empty on track rows.
last_measured_at: Track rows: UTC time of the latest recorded match of any of the row's programs; empty before its first match. Empty on game rows.
first_seen_at: Set only when is_recent_addition is true: the row's last_measured_at in this release, as its first-seen time.
is_recent_addition: true when a row has a rank now and its key had none in the previous comparable cut (absent there, or unranked). The key is the model slug, plus the setting on Reasoning Variants rows. Such a row has no rank_movement.
rank_movement: Places moved in the published ranking since the previous comparable cut: the row's rank there minus its rank now. Positive means the row moved up (a better rank), negative means it moved down, 0 means unchanged. Empty unless the row has a rank in both cuts, so empty for a row not ranked now (including a Reasoning Variants request variant an owner decision lists apart) and for a newly ranked row (see is_recent_addition); empty on every row when the previous cut is not comparable (another game-set version, scoring class, coverage policy or rating method). Both ranks follow the same listing rules, so a rule change on its own, such as a withdrawn model or a request variant listed apart, moves no row.
rank_movement_baseline_at: When the previous rank was measured: the row's last_measured_at in the previous cut, or that cut's identifier if it had none.
score: The headline 0–100 score (see Measures and units). On game rows, working-program strength.
conditional_success_score: Working-program strength of a track row (see Measures and units).
plugin_success_rate: Program success rate from 0 to 1: working programs divided by counted attempts (working programs plus failures in model-written code).
score_basis: How score was computed for this row, in words.
min_score: Track rows: the lowest of the row's per-game headline scores; on Overall rows, the lowest such value among the model's three setting rows.
max_score: Track rows: the highest of the same values.
raw_elo: The B1 rating (see Measures and units). Game rows: the attempt-weighted mean of the working programs' ratings. Track rows: an average of the row's per-game ratings, a cross-game average rather than a rating in one game; it need not equal the mean of the row's game-row raw_elo values. On Overall rows it averages the three settings' per-game ratings and can be empty.
rated_matches: Rated match results of the row's programs, counted per participant; on model-setting rows wins + losses + draws.
attempted_matches: Every recorded match the row's programs took part in, rated or not. The difference from rated_matches is matches that are not match evidence, such as an unconfirmed time-out or an infrastructure failure. Empty on Overall rows.
capped_draws: Of the draws, matches that reached the shared move cap with no fault charged; they count as draws. Empty on Overall rows.
unilateral_faults: Matches with a fault charged to the row's program under the release's fault policy (a crash or illegal move during its move, or a time-out confirmed by a quiet replay); a fault charged to both sides counts for each. Empty on Overall rows.
distinct_rated_opponents: Track rows: opponent rows of the same leaderboard with at least one rated match against this row's programs; on Overall rows, opponent models, the model itself included when its own settings met. Game rows: the sum over the row's programs of each program's distinct rated opponent programs, so an opponent that met two of them counts twice.
wins: Participant wins: the programs' combined record, which measures evidence volume, not strength. Empty when the row has no rated match.
losses: Participant losses, counted like wins.
draws: Participant draws, counted like wins.
uncertainty: Not published under B1: empty on every row (see Measures and units).
entry_count: Track rows: the player programs counted (the Players column): the programs of its game rows; on Overall rows, of its settings.
contributing_entrants: Game rows: the model's programs in that game and setting.
codegen_evaluated_cells: Generation attempts with an outcome: working programs, failures in model-written code and speed-check rejections. Attempts lost to provider, API or test-machine faults are left out.
admitted_plugins: Generation attempts that produced a working player program.
first_pass_admitted: Of those, working on the first try.
repaired_admitted: Of those, working after a repair or resend.
model_code_failures: Attempts that finally failed in model-written code; each counts 0 in the headline.
model_code_failure_rate: model_code_failures divided by admitted_plugins + model_code_failures.
first_pass_rate: first_pass_admitted divided by the same total. The three rates add up to 1.
repaired_rate: repaired_admitted divided by the same total.
infra_failures_excluded: Attempts lost to provider, API or test-machine faults; they count nowhere above.
failed_game_aliases: Games with at least one failure in model-written code (|-separated in CSV).
infra_excluded_game_aliases: Games with at least one attempt lost to such a fault.
performance_rejections: Working programs rejected by the admission speed check: they play no matches and count neither as working programs nor as failures.
performance_rejected_game_aliases: Games with at least one speed-check rejection.
codegen_cost_usd_mean: Estimated US dollars to generate one player program: the equal-weight mean over unique model/game/setting cells, each cell's runs averaged first (see Cost evidence).
codegen_cost_usd_median: Median of those cell means.
codegen_cost_usd_min: Lowest cell mean.
codegen_cost_usd_max: Highest cell mean.
codegen_cost_usd_total: Sum over every generation run with a cost value: workload volume, not a per-program figure.
codegen_cost_unique_cell_count: Unique model/game/setting cells with a cost value, behind the mean (JSON cell_count).
codegen_cost_usd_run_count: Generation runs with a cost value across those cells (JSON run_count). The admitted and failed counts below count the same runs.
codegen_cost_admitted_cell_count: Runs that produced a working program (JSON admitted_cell_count).
codegen_cost_failed_cell_count: Runs that did not, including speed-check rejections (JSON failed_cell_count).
codegen_cost_recorded_attempt_count: Attempts whose record carried a provider-reported or locally estimated dollar value.
codegen_cost_token_priced_attempt_count: Attempts priced from their token counts with a versioned price table.
codegen_cost_legacy_label_count: Attempts whose cost came from an older recorded label.
codegen_cost_basis_counts: JSON object: attempts per cost basis (billed, provider-reported, standard-list estimate, legacy).
codegen_cost_evidence_basis: The cost basis in words (see Cost evidence).
codegen_cost_pricing_table_versions: JSON object: estimates per pricing-table version.
reasoning_bucket: Legacy transport bucket or public setting key; not a comparison key.
reasoning_policy_version: Reasoning-group policy the setting was classified under.
provider_model: The provider's model identifier for the request.
provider_route: Where the request was sent: the provider's own API (for example openai) or openrouter.
requested_reasoning_effort: The same public setting key as reasoning_setting, or the group (baseline, balanced, intensive) where that is empty on a group row; not the literal request.
prompt_identity: Identity of the generation prompt; unknown_legacy when it was not retained.
prompt_profile: Identity of the prompt profile; unknown_legacy when not retained.
code_prompt_version: Version of the code-writing prompt; unknown_legacy when not retained.
code_output_mode: How the program was returned; unknown_legacy when not retained.
generation_policy: quality_generation_policy for benchmark runs. In JSON, generation_policy is an object holding this value and the four timeout fields.
timeout_mode: How generation requests were time-limited; provider_route_timeout_guard is the route's own guard.
stream_idle_timeout_seconds: Idle timeout of a streamed request, when one setting is recorded for the row; empty otherwise.
strict_wall_clock_timeout_seconds: Overall time limit of a request, when recorded; empty otherwise.
streaming_enabled: Whether the request streamed, when recorded; empty otherwise.
generation_wall_clock_seconds: How long generation took, when recorded for one request; empty otherwise.
finish_reason: The provider's finish reason when recorded for one request; unknown_legacy otherwise.
headline_games_json: headline_games as compact JSON (see Measures and units); track rows from scoring v17.
working_attempts: Game rows: working attempts behind the row's programs (see Measures and units).
counted_attempts: Game rows: working attempts plus failures in model-written code.
program_raw_elo_low: Game rows with two or more rated programs: the lowest program rating; not a confidence interval.
program_raw_elo_high: The highest program rating of the same rows.
listing_status: Rows the pages list apart (see Measures and units); empty on every other row.

Row fields only the JSON carries:

evaluated_setting_identity: The evaluated request as an object (CSV evaluated_setting_identity_json).
headline_games: Per-game headline values as an object (CSV headline_games_json).
codegen_cost_usd: Cost evidence as an object (CSV codegen_cost_*), plus cost_is_complete: true only when every cell behind it has complete cost evidence.
reasoning_provenance: The setting's provenance: the CSV's reasoning_bucket, reasoning_policy_version, provider_model, provider_route, requested_reasoning_effort and the four prompt and output identities, plus reasoning_bucket_label, reasoning_bracket, reasoning_catalog_sha256, provider, provider_setting_label (display label), reasoning_request_mode, provider_reasoning_params (untyped request details) and generation_settings_public (temperature, max_tokens, stream, null when not recorded).
generation_policy: An object: generation_policy, timeout_mode and the three timeout and streaming fields of the CSV.
generation_timing: An object: generation_started_at, generation_completed_at, generation_wall_clock_seconds, stream_progress_event_count, first_progress_after_seconds and finish_reason, each null or unknown_legacy when not recorded for one request.

entrants.json holds tracks: for the Reasoning Variants, Baseline, Balanced and Intensive leaderboards (not Overall) and each public game alias, the list of player programs. The Reasoning Variants pool (full_freedom/mixed) lists every program once; the Baseline, Balanced and Intensive lists repeat the programs of their group. Its top level also carries public_release_id, results_cut_id, snapshot_id, benchmark_line, schema_version and generated_at. Each program has these fields:

public_id: Opaque program ID (E-…), stable across releases; the key of publication_readiness.json programs.
model: The model's key, equal to charts_data.json models[].id. It can differ from model_slug (for example gpt-6.1-sol and gpt-6-1-sol); join through Charts.
reasoning: The view key of its reasoning group, as in charts_data.json: none (Baseline), medium (Balanced) or highest (Intensive). Not its provider setting, which is reasoning_setting.
reasoning_setting: The program's recorded provider setting.
reasoning_bracket: Its reasoning group.
reasoning_request_mode: How its request set reasoning (see Reasoning settings).
reasoning_policy_version: Reasoning-group policy its setting was classified under.
reasoning_catalog_sha256: Digest of the capability catalog bound to its request, when retained.
evaluated_setting_identity: Its evaluated request identity (see Row identity and joins).
generated_at: UTC time the program was generated.
repair_status: first_pass (worked on the first try) or repaired (worked after a repair or resend).
coverage_status: rated: it meets the release's match-coverage rule (in current releases, rated matches in both seats against at least 12 opponents, or against every other program of a smaller pool); this is not the full matrix that owed_seat_pairings counts. under_covered: it has rated matches but does not yet meet that rule. pending: no rated match yet. fault_blocked: held back by a recorded fault.
point_rating_method: Rating method of raw_elo_rating; bradley_terry_newton is B1.
confidence_method: Confidence model of a release that publishes program intervals here; absent otherwise.
score: The program's 0–100 score in its game.
raw_elo_rating: Its B1 rating in its game.
uncertainty: Rating uncertainty, published only under earlier rating methods; absent under B1.
posterior_sd_elo: Posterior standard deviation of a confidence model, when one applies; absent otherwise.
confidence_interval_low: Lower bound of that model's interval, when one applies.
confidence_interval_high: Upper bound of the same interval.
wins: Its wins in rated matches.
losses: Its losses in rated matches.
draws: Its draws in rated matches.
rated_matches: Its rated matches: wins + losses + draws.
attempted_matches: Every recorded match it took part in, rated or not (see the leaderboard field).
capped_draws: Its draws that reached the shared move cap.
unilateral_faults: Matches with a fault charged to it.
distinct_rated_opponents: Opponent programs it has a rated match against.
admitted_plugin_count: Generation runs that produced exactly this program; identical code from two runs is one program with a count of 2.
owed_seat_pairings: Seat pairings of the full double round-robin: 2 × (programs in the game's Reasoning Variants pool − 1). A rated program usually has played only some of them.
played_seat_pairings: Of those, pairings with at least one completed rated match.

charts_data.json keys its views inside model entries (for example reasoning, reasoning_labels and official_rank_by_reasoning) as highest (the Intensive leaderboard), medium (Balanced) and none (Baseline); economics, program_outcomes, working_program_strength and game_gaps add all (all settings), and reasoning_settings uses baseline, balanced and intensive. none is the model's lowest tested setting, which is not always reasoning turned off. versus_totals_all holds head-to-head wins, draws and losses between models over all settings, and versus_totals_tracks the same per view; both are keyed by models[].id, which can differ from model_slug. The chart series are described in its schema.

Data dictionary