Methodology

1. The question

What DuelLab measures

DuelLab measures whether an AI model can turn written game rules into a program that plays well. It tests more than whether the code looks plausible: the program must run, make legal decisions, and compete successfully against programs made by other models.

2. Competition

How generated programs compete

Each model receives the rules of a benchmark game and instructions for how its player should communicate with the game. The resulting program is checked and then plays head-to-head matches under the same conditions as its opponents.

Games are deterministic, so the same program, position, and seed produce the same game transition. The evidence set varies starting side, opening position, seeded game setup, and opponent where the game profile defines those dimensions. Those are seat-, setup-, or opponent-specific results—not lucky repeats of an identical match.

3. Evaluator ceiling

Why GameBench 2 has no evaluator ceiling

GameBench 2 does not ask a human or AI judge to decide whether one program looks better or played more intelligently. The deterministic engine applies the game rules and records wins, losses, draws, illegal moves, and execution failures.

Those outcomes feed the declared rating and scoring policy. The benchmark therefore does not need an evaluator smarter than the system being tested: it can continue comparing human-level, superhuman, AGI, and ASI performance through direct competition.

This describes the evaluation mechanism, not the current leaderboard. A score does not by itself prove that a system is human-level, superhuman, AGI, or ASI. Scores are relative to the evaluated field, and any fixed game can eventually saturate; DuelLab can add games and evidence as the field advances.

4. Reasoning

Why reasoning settings rank separately

A model can produce a different player program when its reasoning setting changes. DuelLab groups the tested settings into Baseline, Balanced, and Intensive. These groups are relative to each model: Baseline is its lowest tested setting, Balanced is a middle setting, and Intensive is its highest tested setting. They are not equal amounts of work across different providers.

The Reasoning Variants leaderboard therefore gives each model and tested setting its own row. The exact provider setting is shown beside the public group. Names that refer to the same request are combined; genuinely different requests remain separate.

5. Scoring

How match results become a 0–100 score

Wins, losses, and draws produce a B1 (Bradley–Terry) rating for each program in each game, and the rating becomes a 0–100 score: 100 × the program's expected points against the fixed reference field. Stronger results receive higher scores.

When a model produced more than one working player program for the same game and setting, DuelLab combines those results first. The Playable score is the average across games where the model produced a working program. The headline score counts every generation attempt: each attempt that failed for a model cause adds a 0 to its game beside the working programs.

This is a relative score against a fixed reference field of player programs, not a percentage of games solved. It can move when match evidence or games change, because the field's ratings are refitted at every release, or when a new field version is announced; models added later are scored against the field without joining it. Ordinary match results and draws caused by a shared move limit count. A program that crashes during its move or makes an illegal move loses that match. Timeouts, other one-sided faults and infrastructure failures do not count as match evidence. When an earlier version of the scheduler replayed a match after a crash, between the same two programs in the same seats and under the same match conditions, only the first result counts; the replays stay recorded as played matches. Planned repeat matches count normally.

Three different kinds of evidence

Match evidence: opponent coverage, starting-side coverage and recorded matches describe the retained programs. Under B1 a program's rating carries no uncertainty term, so no rating-evidence index is shown. Opponents played, an optional leaderboard column that is hidden by default, measures progress through the full matrix. The benchmark owes each player program one rated match against every other program in the same game's pool with each of the two moving first: 2 × (pool − 1) seat pairings. A program's share is the number of those seat pairings with at least one completed rated match divided by the number owed. A row pools its programs (played pairings summed, divided by owed pairings summed); an Overall row pools the model's programs from its Baseline, Balanced and Intensive settings. The pool is the game's Reasoning Variants pool, which the other views filter after rating. Releases whose data was produced before October 8, 2026 have no seat-level counts; they owe each opponent once and count it as played after any completed rated match against it. It describes how far the schedule has progressed, not how precise a rating is.

Generation evidence: contributing player counts and generation outcomes describe the programs produced. More matches with the same programs do not add independent generations. This release does not estimate a confidence interval for the model's next generated program.

Cross-game variation: inspect the per-game scores and range separately. A high average can hide weaker games. B1's rating has no published interval in this release.

6. Missing results

How failures and unevaluated games are treated

A final failure in model-written code counts as 0 in the headline score (scoring policy v17): each failed attempt sits beside the game's working programs, and a reasoning setting whose every attempt failed scores 0. It is not turned into a match result, so no opponent gains rating from it. A repair or resend does not lower the score when it eventually produces a working player program, but it remains visible in Codegen and estimated cost.

A working program that is too slow for the admission speed check is disqualified: it plays no matches, and its game is left out of Playable. Since October 13, 2026 (scoring policy v16) this rejection does not lower the headline score either; since v17 the rejected attempt counts as neither a working program nor a failure of its game, and a game with no other evaluated attempt is left out of the headline mean. Earlier releases counted it as a failure.

If a model has not yet been evaluated on a game, that absence is not silently counted as a loss. The standings and detailed tables distinguish missing evidence from a known failed attempt.

The Reasoning Variants leaderboard gives an official rank only after a row has an evaluated generation attempt in every one of the eight games (a working program, a failed attempt or a speed-check rejection; a repeat attempt in one game does not stand in for another game), enough working programs, and match coverage. A row that is still below that requirement stays visible as Provisional and receives no rank.

Some games cannot be scored: when a game's only generation attempts failed because of our test harness or the provider, not the model, and cannot currently be repeated, the game has no value for that row. The row's score averages its other games, and the row stays Provisional. A reasoning setting whose every generation attempt failed is published at 0; it stays Provisional too, and counts as one of the model's three settings in Overall.

The Overall leaderboard ranks only models with scores for all three reasoning settings (since October 13, 2026). An average of fewer settings leaves out the weaker ones and is not comparable, so those models are listed below it without an Overall rank; each of their settings is ranked in the Model + reasoning setting view. A Partial 3/3 row has all three scores but another evidence or quality check is still missing; the page shows the reason beside the rank.

7. Codegen health

What first-pass, repaired, failed, and n mean

Codegen shows how often a model produced a working player program on the first try, produced one after a repair or resend, or still failed. Its n is the number of model runs included, so repeated evaluations remain visible in the total.

A program that cannot be read, checked, compiled, started, or run within the benchmark limits counts as a failure in model-written code. Provider, login, rate-limit, timeout, and other infrastructure failures are shown separately and do not enter n or the percentages. Neither does a program rejected only by the speed check; it is listed separately.

8. Cost evidence

How code-generation cost is calculated

Std. price is an estimated generation-and-repair cost. Each unique combination of model, game, and reasoning setting receives equal weight. Repeated runs of the same combination are averaged first, so rerunning one combination cannot give it extra influence. The median estimate and total recorded cost are shown alongside the mean.

DuelLab uses a reconciled bill when available, then a cost reported by the provider, then a versioned estimate from public prices, and finally a clearly labeled older estimate. The page identifies which basis was used. A measured free or zero-cost run remains zero instead of being treated as missing.

9. Updates

Why standings change over time

GameBench 2 is continuously updated. New models and games can be added, and existing rows can gain more match evidence or be refreshed. These changes can move scores and ranks even when a particular model has not changed.

Use the public Updated date to identify the current standings, and treat close scores as close evidence rather than a permanent ordering.

10. Benchmark generations

GameBench 2 and the frozen GameBench 1 archive

GameBench 2 is the evolving benchmark used for current standings. GameBench 1 was designed as a frozen historical benchmark: its games and rankings are preserved as an archive rather than extended with new models.

The two benchmarks use different game sets and evaluation contexts, so their scores and ranks are not directly comparable. Open the GameBench 1 archive.

11. Interpretation

Limits and responsible interpretation

DuelLab measures one demanding ability: creating programs that play the included games well. It does not measure every aspect of coding, reasoning, safety, usefulness, or real-world software development.

Small score differences may reflect limited evidence. Compare match counts, individual games, reasoning settings, and known failures before drawing broad conclusions about a model.

12. Execution safety

How generated code is treated

DuelLab treats every model-written player program as untrusted. Programs are built and run with limits on network access, files, child processes, time, CPU, and memory. Exact security settings are not published because they could help someone work around them. Release records keep game results separate from failures caused by the host, compiler, scheduler, or player program.

13. Evidence checkpoints

What the evidence badges mean

Every program is meant to play every program from other models, in both seats. That full schedule takes a long time, so DuelLab also marks results that have reached an early, representative checkpoint.

A Preliminary badge means 48 opponents were drawn for the program at random, in proportion to how much each counts in the benchmark's target, and every drawn pairing has been played in both seats. The badge says the evidence is representative enough to show. It is not a confidence interval or a precision claim, and it does not change scores or ranks. Program badges appear only beside rated rows. Model and setting rows show no such badge; their states are in the data file.

Each badge names the target population it was earned under; hover over it for the revision and the evidence position. When new programs join, the target changes. A badge earned under an earlier population stays, shows that revision (for example rev 3f9a2c1d), and its details say where the result stands under the current target. Programs whose target includes a pairing that ended in an unscoreable fault receive no new badge. The states are also published in publication_readiness.json.

An Interval validated badge adds a 95% interval for the program's target-weighted pairing score, from 0 to 1 (not the 0–100 score). It is estimated from the program's 48 drawn pairings with the help of a rating model: drawn pairings keep their results, and the pairings not drawn are predicted from the program's fitted strength. Hover over it for the estimate and its width. The method was validated across programs on 14 October 2026: on a sample of programs whose full schedule had been played, 96.8% of these intervals contained the result of the full schedule, and at least 93.7% in every game. In Chess the intervals are wider than they need to be (99.7% contained it). That describes the method, not each program; a particular interval can still miss. The validation carries over across engine and SDK upgrades, including rule corrections: results played under different versions are pooled, and the method is not validated again after each upgrade. This release gives intervals for individual programs only; model and setting rows get none until an interval for whole rows has been validated.

An Interval validated badge appears only for an interval method this release's state policy qualifies. A badge earned under an earlier target with another method is shown as Preliminary when its interval came from the 48-draw checkpoint, and is not shown otherwise.

A few programs cannot be scored reliably: the benchmark worker has quarantined them, and a quarantine lasts until an operator lifts it, so pairings with them would keep other programs from completing a checkpoint. Those programs leave every other program's target, and the remaining weights are scaled up proportionally. A crash or an illegal move does not remove a program: it counts as that program's loss. The list is fixed for each target population revision and counted in the data file.