Methodology

1. The question

What DuelLab measures

DuelLab measures whether an AI model can turn written game rules into a program that plays well. It tests more than whether the code looks plausible: the program must run, make legal decisions, and compete successfully against programs made by other models.

2. Competition

How generated programs compete

Each model receives the rules of a benchmark game and instructions for how its player should communicate with the game. The resulting program is checked and then plays head-to-head matches under the same conditions as its opponents.

Games are deterministic, so the same program, position, and seed produce the same game transition. The evidence set varies starting side, opening position, seeded game setup, and opponent where the game profile defines those dimensions. Those are seat-, setup-, or opponent-specific results—not lucky repeats of an identical match.

3. Evaluator ceiling

Why GameBench 2 has no evaluator ceiling

GameBench 2 does not ask a human or AI judge to decide whether one program looks better or played more intelligently. The deterministic engine applies the game rules and records wins, losses, draws, illegal moves, and execution failures.

Those outcomes feed the declared rating and scoring policy. The benchmark therefore does not need an evaluator smarter than the system being tested: it can continue comparing human-level, superhuman, AGI, and ASI performance through direct competition.

This describes the evaluation mechanism, not the current leaderboard. A score does not by itself prove that a system is human-level, superhuman, AGI, or ASI. Scores are relative to the evaluated field, and any fixed game can eventually saturate; DuelLab can add games and evidence as the field advances.

4. Reasoning

Why reasoning settings rank separately

A model can produce a different player program when its reasoning setting changes. DuelLab groups the tested settings into Baseline, Balanced, and Intensive. These groups are relative to each model: Baseline is its lowest tested setting, Balanced is a middle setting, and Intensive is its highest tested setting. They are not equal amounts of work across different providers.

The Reasoning Variants leaderboard therefore gives each model and tested setting its own row. The exact provider setting is shown beside the public group. Names that refer to the same request are combined; genuinely different requests remain separate.

5. Scoring

How match results become a 0–100 score

Wins, losses, and draws produce a rating for each program in each game. DuelLab subtracts an allowance for rating uncertainty and converts the result to a 0–100 score; stronger results receive higher scores.

When a model produced more than one working player program for the same game and setting, DuelLab combines those results first. The Playable score is the average across games where the model produced a working program. The headline score also accounts for how often program generation succeeded.

This is a relative score against the current models, not a percentage of games solved. It can move when models, games, or evidence change. Ordinary match results and draws caused by a shared move limit count; one-sided player faults and infrastructure failures do not count as match evidence.

Three different kinds of evidence

Match evidence: opponent coverage, starting-side coverage, recorded matches, and the rating-evidence index describe the retained programs. The displayed index is a heuristic: clamp(100 × (u − 35) / 80, 0, 100), with u the program's raw rating uncertainty: 400 / sqrt(n + 1) under sequential Elo, or the rating deviation (RD) under the Glicko-style rating. Zero means u is at or below 35; 100 means u is at or above 115. The 35 and 115 anchors were set for the Elo penalty, so under the Glicko-style rating, whose RD floor is 30, the index is usually 0. Zero does not mean certainty. The aggregate match-evidence index (formerly Signal) is clamp(100 − mean displayed uncertainty index, 0, 100): higher means more match evidence under this heuristic, not a probability of correctness. A value of 100 can therefore coexist with unresolved generation variability.

Generation evidence: contributing player counts and generation outcomes describe the programs produced. More matches with the same programs do not add independent generations. This release does not estimate a confidence interval for the model's next generated program.

Cross-game variation: inspect the per-game scores and range separately. A high average can hide weaker games. A 90% rating band, when supplied by an alternative rating estimator, concerns its fitted rating and is transformed onto the current score scale; it does not cover generation-to-generation variation. The Glicko-style estimator does not supply that interval.

6. Missing results

How failures and unevaluated games are treated

A final failure in model-written code lowers the headline score gradually; DuelLab does not turn it into a pretend zero-strength match result. A repair or resend does not lower the score when it eventually produces a working player program, but it remains visible in Codegen and estimated cost.

If a model has not yet been evaluated on a game, that absence is not silently counted as a loss. The standings and detailed tables distinguish missing evidence from a known failed attempt.

The Reasoning Variants leaderboard gives an official rank only after a row has enough working programs and match coverage. A row that is still below that requirement stays visible as Provisional and receives no rank.

The Overall leaderboard ranks every model with at least one available reasoning-group score. A Partial n/3 label means that the official rank is based on incomplete evidence, so it should be compared cautiously. A row can be Partial 3/3 when all three scores exist but another evidence or quality check is still missing; the page shows the reason beside the rank.

7. Codegen health

What first-pass, repaired, failed, and n mean

Codegen shows how often a model produced a working player program on the first try, produced one after a repair or resend, or still failed. Its n is the number of model runs included, so repeated evaluations remain visible in the total.

A program that cannot be read, checked, compiled, started, or run within the benchmark limits counts as a failure in model-written code. Provider, login, rate-limit, timeout, and other infrastructure failures are shown separately and do not enter n or the percentages.

8. Cost evidence

How code-generation cost is calculated

Std. price is an estimated generation-and-repair cost. Each unique combination of model, game, and reasoning setting receives equal weight. Repeated runs of the same combination are averaged first, so rerunning one combination cannot give it extra influence. The median estimate and total recorded cost are shown alongside the mean.

DuelLab uses a reconciled bill when available, then a cost reported by the provider, then a versioned estimate from public prices, and finally a clearly labeled older estimate. The page identifies which basis was used. A measured free or zero-cost run remains zero instead of being treated as missing.

9. Updates

Why standings change over time

GameBench 2 is continuously updated. New models and games can be added, and existing rows can gain more match evidence or be refreshed. These changes can move scores and ranks even when a particular model has not changed.

Use the public Updated date to identify the current standings, and treat close scores as close evidence rather than a permanent ordering.

10. Benchmark generations

GameBench 2 and the frozen GameBench 1 archive

GameBench 2 is the evolving benchmark used for current standings. GameBench 1 was designed as a frozen historical benchmark: its games and rankings are preserved as an archive rather than extended with new models.

The two benchmarks use different game sets and evaluation contexts, so their scores and ranks are not directly comparable. Open the GameBench 1 archive.

11. Interpretation

Limits and responsible interpretation

DuelLab measures one demanding ability: creating programs that play the included games well. It does not measure every aspect of coding, reasoning, safety, usefulness, or real-world software development.

Small score differences may reflect limited evidence. Compare match counts, individual games, reasoning settings, and known failures before drawing broad conclusions about a model.

12. Execution safety

How generated code is treated

DuelLab treats every model-written player program as untrusted. Programs are built and run with limits on network access, files, child processes, time, CPU, and memory. Exact security settings are not published because they could help someone work around them. Release records keep game results separate from failures caused by the host, compiler, scheduler, or player program.

13. Evidence checkpoints

What the evidence badges mean

Every program is meant to play every program from other models, in both seats. That full schedule takes a long time, so DuelLab also marks results that have reached an early, representative checkpoint.

A Preliminary badge means 48 opponents were drawn for the program at random, in proportion to how much each counts in the benchmark's target, and every drawn pairing has been played in both seats. A model and setting earns the badge when the same holds for its programs in every game; per-game states are in the data file. The badge says the evidence is representative enough to show. It is not a confidence interval or a precision claim, and it does not change scores or ranks. Program badges appear only beside rated rows.

Each badge names the target population it was earned under; hover over it for the revision and the evidence position. When new programs join, the target changes. A badge earned under an earlier population stays, shows that revision (for example rev 3f9a2c1d), and its details say where the result stands under the current target. Programs whose target includes a pairing that ended in an unscoreable fault receive no new badge. The states are also published in publication_readiness.json.

This release shows no Interval validated badge. Its results pool matches played under more than one SDK version, while the interval method was validated on results from a single version, so the intervals return once the method has been validated again. The Preliminary badge does not depend on that validation and is shown as usual.

A few programs cannot be scored reliably: they were permanently quarantined, or crashed or made illegal moves in at least two matches, so pairings with them would keep other programs from completing a checkpoint. Those programs leave every other program's target, and the remaining weights are scaled up proportionally. The list is fixed for each target population revision and counted in the data file.