This copy of the methodology was frozen when release gb2-20261119-235af8c04037 was published on . Later changes appear only in the current methodology.

How a result is made

  1. Rules

    Each model receives a game's rules and the interface its player must use.

  2. Player program

    The model writes a program. DuelLab checks it, then builds and runs it within fixed limits.

  3. Matches

    Programs from different models play head-to-head, varying starting side and setup. The engine applies the rules.

  4. Scores and ranks

    Wins, draws, and losses become a 0–100 score for each game, then the leaderboard ranking.

What it measures

What DuelLab measures

DuelLab measures whether an AI model can turn written game rules into a program that plays well. It tests more than whether the code looks plausible: the program must run, make legal decisions, and compete successfully against programs made by other models.

Read more

The GameBench 2 technical report explains the method in more depth and analyses the 20 October 2026 release. It predates later method changes, such as the move time-out rule; this page is current.

How generated programs compete

Each model receives the rules of a benchmark game and instructions for how its player should communicate with the game. The resulting program is checked and then plays head-to-head matches under the same conditions as its opponents.

Read more

Games contain no hidden randomness: the same programs, from the same starting position and seed, always play out the same way. So matches are varied on purpose: starting side, opening position, seeded setup, and opponent, where the game has them. Each result is a different test, not a lucky repeat of the same match.

Why GameBench 2 has no evaluator ceiling

GameBench 2 does not ask a human or AI judge to decide whether one program looks better or played more intelligently. The deterministic engine applies the game rules and records wins, losses, draws, illegal moves, and execution failures. So scoring is never limited by how good a judge is.

Read more

Those outcomes feed the declared rating and scoring policy. The benchmark therefore does not need an evaluator smarter than the system being tested: it can continue comparing human-level, superhuman, AGI, and ASI performance through direct competition.

A score does not by itself prove that a system is human-level, superhuman, AGI, or ASI. Scores are relative to the evaluated field, and any fixed game can eventually saturate; DuelLab can add games and evidence as the field advances.

Why reasoning settings rank separately

A model can produce a different player program when its reasoning setting changes. DuelLab groups the tested settings into Baseline, Balanced, and Intensive. These groups are relative to each model: Baseline is its lowest tested setting, Balanced is a middle setting, and Intensive is its highest tested setting. They are not equal amounts of work across different providers.

Read more

The Reasoning Variants leaderboard therefore gives each model and tested setting its own row. The exact provider setting is shown beside the public group. Names that refer to the same request are combined; genuinely different requests remain separate.

How scores are made

How match results become a 0–100 score

Wins, losses, and draws give each program a strength rating in each game (the method is B1, a Bradley–Terry rating). The rating becomes a 0–100 score: 100 × the share of points the program would be expected to take against a fixed set of reference programs, with a win counting 1 and a draw ½. Stronger results receive higher scores.

Read more

When a model produced more than one working player program for the same game and setting, DuelLab combines those results first. Charts and tables use two scores:

  • Working-program strength (the leaderboard's Strength column): the average across games of the working programs alone; failed generation attempts are left out.
  • Headline score (the public score that ranks models): counts every generation attempt; each attempt that failed for a model cause adds a 0 to its game beside the working programs.

The program success rate is shown on its own. When a model has several working programs for one game and setting, that row's strength and raw rating are their average, weighted by how many generation attempts produced each, and its win/draw/loss record is their combined total: it shows how much was played, not how strong the row is.

This is a relative score against a fixed set of reference player programs (the reference field), not a percentage of games solved. It can move when match evidence or games change, because the reference programs' ratings are refitted at every release, or when a new version of the reference field is announced; models added later are scored against it without joining it.

Ordinary match results and draws caused by a shared move cap count. A program that crashes during its move or makes an illegal move loses that match. Running out of time on a move is not an immediate loss: the same pairing is replayed at least two hours later on an otherwise quiet machine, and only if the same program runs out of time again while it is computing does it lose; until then the timed-out match does not count. Other one-sided faults and infrastructure failures do not count as match evidence.

When an earlier version of the scheduler replayed a match after a crash, between the same two programs in the same seats and under the same match conditions, only the first result counts; the replays stay recorded as played matches. Planned repeat matches count normally.

Technical specification

The public per-game estimator is B1, a Bradley–Terry rating fitted to convergence. All rated matches in the game are fitted together, so the order in which matches were played does not matter; outcomes are 1 for a win, 0.5 for a draw, and 0 for a loss. The fit includes a first-player term and a weak prior that pulls every strength towards the centre of the game's programs (precision 0.25 per strength, in logits), so a program that never lost or never won still has a finite rating. Newton's method runs until its step is below 10−10. The rating has no rating deviation and nothing is deducted from it; it is shown on a rating scale centred near 1500.

A program's 0–100 score is 100 × its expected points (a win 1, a draw ½) against a fixed reference field, with both seats averaged. The field holds every model with equal total weight, split inside each model by setting, request cohort and program, each distinct program once; a program in the field meets itself at ½. Because every program faces the same field, scores keep the order of the ratings, and one program joining or leaving moves the others very little. The field's members and weights stay fixed within a field version, recorded with the release; the strengths are refitted at every release. A member whose program no longer plays new matches (for example because it does not compile under a newer SDK) keeps its place while it still has a rating; only a member that loses its rating starts a new field version. A model's game value counts its working programs and each failed generation attempt; the headline score is the equal-weighted mean across games, in which each game combines its working programs' scores with a 0 for every program generation attempt that failed for a model cause, and a game lost only to the harness is left out; the program success rate is published on its own.

B1 replaced the Glicko-style rating after a pre-specified comparison on 14 October 2026: on held-out pairings it predicted results no worse than the Glicko-style kernel, and unlike that kernel it converges instead of depending on a fixed number of passes. Ratings and scores from these releases are not directly comparable with earlier releases.

Three different kinds of evidence

Match evidence: opponent coverage, starting-side coverage and recorded matches describe the retained programs. The B1 rating has no margin of error, so no rating-evidence index is shown. Opponents played, an optional leaderboard column that is hidden by default, measures progress through the full matrix. Each player program is due one rated match against every other program in the same game's pool, once moving first and once moving second: 2 × (pool − 1) seat pairings. A program's share is the number of those seat pairings with at least one completed rated match, divided by the number due. A row pools its programs (played pairings summed, divided by due pairings summed); an Overall row pools the model's programs from its Baseline, Balanced and Intensive settings. The pool is every program of the game in the Reasoning Variants leaderboard; the other views are filtered from it after rating. Releases whose data was produced before October 8, 2026 have no seat-level counts; they expect one match per opponent and count it as played after any completed rated match against it. It describes how far the schedule has progressed, not how precise a rating is.

Generation evidence: contributing player counts and generation outcomes describe the programs produced. More matches with the same programs do not add independent generations. This release does not estimate a confidence interval for the model's next generated program.

Cross-game variation: inspect the per-game scores and range separately. A high average can hide weaker games. B1's rating has no published interval in this release.

How failures and unevaluated games are treated

When a model's attempt to write a player program finally fails because of its own code, that attempt counts as 0 for its game in the headline score, alongside the scores of the game's working programs. A reasoning setting whose every attempt failed scores 0. A failure is not turned into a match result, so no opponent gains rating from it. A game the model has not been evaluated on yet is not counted as a loss. A repair or resend does not lower the score when it eventually produces a working player program, but it remains visible in Codegen and estimated cost.

Read more

A working program that is too slow for the admission speed check is disqualified: it plays no matches, and its game is left out of working-program strength. It does not lower the headline score either: the rejected attempt counts as neither a working program nor a failure, and a game with no other evaluated attempt is left out of the headline mean. Releases before October 13, 2026 counted it as a failure.

The standings and detailed tables distinguish a game not yet evaluated from a known failed attempt.

The Reasoning Variants leaderboard gives an official rank only after a row has an evaluated generation attempt in every one of the eight games (a working program, a failed attempt or a speed-check rejection; a repeat attempt in one game does not stand in for another game), enough working programs, and match coverage. A row that is still below that requirement stays visible as Provisional and receives no rank.

When a game's only generation attempts failed because of our test setup or the provider, not the model, the game has no value for that row until a new attempt succeeds. The row's score averages its other games, and the row stays Provisional. Where no new attempt can be made, because the provider route no longer exists or a new request would not reproduce the original setting, the row is marked Can't be ranked in the Not ranked list. Sometimes the same model and reasoning setting was requested in two ways (two request variants). If one is ranked and the other is Provisional only because games are missing, the Provisional one is not listed; if both are complete, a recorded decision may list only one, and the ranks have no gap. The downloads keep each such row (listing_status), and its programs still count toward the model's Overall score. A reasoning setting whose every generation attempt failed is published at 0; it stays Provisional too, and counts as one of the model's three settings in Overall.

The Overall leaderboard ranks only models with scores for all three reasoning settings (since October 13, 2026). An average of fewer settings leaves out the weaker ones and is not comparable, so those models are listed below it without an Overall rank; each of their settings is ranked in the Reasoning Variants leaderboard (Model + reasoning setting). A row marked Partial 3/3 has all three setting scores (3/3) but is partial because another evidence or quality check is still missing; the page shows the reason beside the rank.

What first-pass, repaired, failed, and n mean

Codegen shows how often a model produced a working player program on the first try, produced one after a repair or resend, or still failed. Its n is the number of model runs included, so repeated evaluations remain visible in the total.

Read more

A program that cannot be read, checked, compiled, started, or run within the benchmark limits counts as a failure in model-written code. Provider, login, rate-limit, timeout, and other infrastructure failures are shown separately and do not enter n or the percentages. Neither does a program rejected only by the speed check; it is listed separately.

How code-generation cost is calculated

Cost / player is the estimated cost of generating and repairing one player program. Each unique combination of model, game, and reasoning setting receives equal weight. Repeated runs of the same combination are averaged first, so rerunning one combination cannot give it extra influence. The median estimate and total recorded cost are shown alongside the mean.

Read more

DuelLab uses a reconciled bill when available, then a cost reported by the provider, then a versioned estimate from public prices, and finally a clearly labeled older estimate. The page identifies which basis was used. Because the basis can differ between models, costs compare approximately; they are not one uniform price list. A measured free or zero-cost run remains zero instead of being treated as missing.

What the evidence badges mean

Every program is meant to play every program from other models, in both seats. That full schedule takes a long time, so DuelLab also marks results that have reached an early, representative checkpoint. The checkpoint draws its opponents from the program's target: the set of opponents it is meant to face, each weighted by how much it counts in the benchmark.

Read more

A Preliminary badge means 48 random draws of opponents were made for the program (an opponent can be drawn more than once), in proportion to each opponent's weight in the target, and every drawn pairing has been played in both seats. The badge says the evidence is representative enough to show. It is not a confidence interval or a precision claim, and it does not change scores or ranks. Program badges appear only beside rated rows. Model and setting rows show no such badge; their states are in the data file.

Each badge names the version of the target it was earned under; hover over it for that version and the point in the evidence where it was earned. When new programs join, the target changes. A badge earned under an earlier target stays and shows its version (for example rev 3f9a2c1d); its details say where the result stands under the current target. Programs whose target includes a pairing that ended in an unscoreable fault receive no new badge. The states are also published in publication_readiness.json.

An Interval validated badge adds a 95% interval for the program's average result against its target opponents, weighted as in the target, on a 0–1 scale (a different number from the 0–100 score). It is estimated from the program's 48 drawn pairings with the help of a rating model: drawn pairings keep their results, and the pairings not drawn are predicted from the program's fitted strength. Hover over it for the estimate and its width. The method was validated across programs on 14 October 2026: on a sample of programs whose full schedule had been played, 96.8% of these intervals contained the result of the full schedule, and at least 93.7% in every game. In Chess the intervals are wider than they need to be (99.7% contained it). That describes the method, not each program; a particular interval can still miss. The validation carries over across engine and SDK upgrades, including rule corrections: results played under different versions are pooled, and the method is not validated again after each upgrade. This release gives intervals for individual programs only; model and setting rows get none until an interval for whole rows has been validated.

An Interval validated badge is shown only for an interval method this release accepts as validated. A badge earned under an earlier target with another method is shown as Preliminary if its interval came from the 48-draw checkpoint, and is hidden otherwise.

A few programs cannot be scored reliably: the benchmark worker has quarantined them, and a quarantine lasts until an operator lifts it, so pairings with them would keep other programs from completing a checkpoint. Those programs leave every other program's target, and the remaining weights are scaled up proportionally. A crash or an illegal move does not remove a program: it counts as that program's loss. The list is fixed for each target population revision and counted in the data file.

Technical specification

Draws follow qualification_draws_v1: with replacement, in proportion to target weight, defined by SHA-256 and exact rational arithmetic. A repeated draw reuses its pairing, so 48 draws usually cover 38–47 distinct opponents. An Interval validated state, with a 95% interval for the program's target-weighted score, is shown only for an interval method, unit and checkpoint that passed a predeclared validation; that interval describes the target-weighted score, not the 0–100 score. Validation says nothing about width: the badge adds its half-width as a number, and the data file also records whether it is within 0.05. The validated method is bt_newton_model_assisted at 48 draws, with the program as the unit (validation of 14 October 2026, evidence cut 1023737, results pooled from three versions). Opponent strengths come from a Bradley–Terry fit (prior precision 0.25 per logit, with a first-player term) of every scored match of the game except the program's own, and the program's strength is fitted from its distinct drawn pairings against those fixed strengths. Drawn pairings keep their observed score and the rest of the target is predicted; the variance is the delta method on the strength's posterior variance plus the predicted pairings' outcome variance. Coverage was 0.968 (program bootstrap 0.961–0.975) and at least 0.937 in every game; Chess overcovers (0.997), so its intervals are wider than they need to be. It replaces direct_hh_normal (validated on 4 October 2026), which covered 0.931 (0.920–0.939) in the same test, so its 93% requirement is no longer demonstrated. That validation carries over across engine and SDK upgrades, so it also covers results pooled across versions; the data file's interval_validation records it. Model and setting rows have no interval badge until an interval for whole rows has been validated. Target exclusions follow readiness_target_exclusions_v2: a program is excluded only while the worker's quarantine on it is active.

Changes over time

Why standings change over time

GameBench 2 is continuously updated. New models and games can be added, and existing rows can gain more match evidence or be refreshed. Each release also recomputes every program's rating from all rated matches so far. These changes can move scores and ranks even when a particular model has not changed.

Read more

After a rules or engine correction, programs replay their matches under the new version. Only the newest version's results of each pairing count; older results stay in the data. Until every pair is replayed, a game mixes versions.

Use the public Updated date to identify the current standings, and treat close scores as close evidence rather than a permanent ordering.

GameBench 2 and the frozen GameBench 1 archive

GameBench 2 is the evolving benchmark used for current standings. GameBench 1 was designed as a frozen historical benchmark: its games and rankings are preserved as an archive rather than extended with new models.

Read more

The two benchmarks use different game sets and evaluation contexts, so their scores and ranks are not directly comparable. Open the GameBench 1 archive.

Using the results

Limits and responsible interpretation

DuelLab measures one demanding ability: creating programs that play the included games well. It does not measure every aspect of coding, reasoning, safety, usefulness, or real-world software development.

Read more

Small score differences may reflect limited evidence. Compare match counts, individual games, reasoning settings, and known failures before drawing broad conclusions about a model.

How generated code is treated

DuelLab treats every model-written player program as untrusted. Programs are built and run with limits on network access, files, child processes, time, CPU, and memory. Exact security settings are not published because they could help someone work around them. Release records keep game results separate from failures caused by the host, compiler, scheduler, or player program.

Cite the exact results you used

Use duellab.org/citation when citing DuelLab. To make your source reproducible, include the release ID, full UTC generation time, benchmark version, scoring rules, rating method, and permanent release link from release-manifest.json.

Read more

Ready-to-use files are available for this generated release as BibTeX, CSL-JSON, RIS, CITATION.cff, and plain text.

Public benchmark JSON, CSV, and derived chart data are licensed under CC BY 4.0. Credit DuelLab, link the license, indicate changes, and do not imply endorsement.

Machine-readable results

Researchers who want to inspect or reproduce their own comparisons can download the current public standings as JSON (an index of per-leaderboard files) or CSV, Parquet, and player-program JSON. The data dictionary defines every leaderboard and player-program field and explains missing values, units, and the connections between the files. Scripts must send their own User-Agent; it shows how.