Methodology

1. The question

What DuelLab measures

DuelLab measures whether an AI model can turn written game rules into a program that plays well. It tests more than whether the code looks plausible: the program must run, make legal decisions, and compete successfully against programs made by other models.

2. Competition

How generated programs compete

Each model receives the rules of a benchmark game and instructions for how its player should communicate with the game. The resulting program is checked and then plays head-to-head matches under the same conditions as its opponents.

Games are deterministic, so the same program, position, and seed produce the same game transition. The evidence set varies starting side, opening position, seeded game setup, and opponent where the game profile defines those dimensions. Those are seat-, setup-, or opponent-specific results—not lucky repeats of an identical match.

3. Reasoning

Why reasoning settings rank separately

A model can produce a different player program when its reasoning setting changes. DuelLab groups the tested settings into Baseline, Balanced, and Intensive. These groups are relative to each model: Baseline is its lowest tested setting, Balanced is a middle setting, and Intensive is its highest tested setting. They are not equal amounts of work across different providers.

The Reasoning Variants leaderboard therefore gives each model and tested setting its own row. The exact provider setting is shown beside the public group. Names that refer to the same request are combined; genuinely different requests remain separate.

4. Scoring

How match results become a 0–100 score

Wins, losses, draws, and the amount of evidence available produce a score for each game. The score is shown on a 0–100 scale, with stronger and more certain performance receiving a higher value.

When a model produced more than one working player program for the same game and setting, DuelLab combines those results first. The Playable score is the average across games where the model produced a working program. The headline score also accounts for how often program generation succeeded.

This is a relative score against the current models, not a percentage of games solved. It can move when models, games, or evidence change. Ordinary match results and draws caused by a shared move limit count; one-sided player faults and infrastructure failures do not count as match evidence.

5. Missing results

How failures and unevaluated games are treated

A final failure in model-written code lowers the headline score gradually; DuelLab does not turn it into a pretend zero-strength match result. A repair or resend does not lower the score when it eventually produces a working player program, but it remains visible in Codegen and estimated cost.

If a model has not yet been evaluated on a game, that absence is not silently counted as a loss. The standings and detailed tables distinguish missing evidence from a known failed attempt.

The Reasoning Variants leaderboard gives an official rank only after a row has enough working programs and match coverage. A row that is still below that requirement stays visible as Provisional and receives no rank.

The Overall leaderboard ranks every model with at least one available reasoning-group score. A Partial n/3 label means that the official rank is based on incomplete evidence, so it should be compared cautiously. A row can be Partial 3/3 when all three scores exist but another evidence or quality check is still missing; the page shows the reason beside the rank.

6. Codegen health

What first-pass, repaired, failed, and n mean

Codegen shows how often a model produced a working player program on the first try, produced one after a repair or resend, or still failed. Its n is the number of model runs included, so repeated evaluations remain visible in the total.

A program that cannot be read, checked, compiled, started, or run within the benchmark limits counts as a failure in model-written code. Provider, login, rate-limit, timeout, and other infrastructure failures are shown separately and do not enter n or the percentages.

7. Cost evidence

How code-generation cost is calculated

Std. price is an estimated generation-and-repair cost. Each unique combination of model, game, and reasoning setting receives equal weight. Repeated runs of the same combination are averaged first, so rerunning one combination cannot give it extra influence. The median estimate and total recorded cost are shown alongside the mean.

DuelLab uses a reconciled bill when available, then a cost reported by the provider, then a versioned estimate from public prices, and finally a clearly labeled older estimate. The page identifies which basis was used. A measured free or zero-cost run remains zero instead of being treated as missing.

8. Updates

Why standings change over time

GameBench 2 is continuously updated. New models and games can be added, and existing rows can gain more match evidence or be refreshed. These changes can move scores and ranks even when a particular model has not changed.

Use the public Updated date to identify the current standings, and treat close scores as close evidence rather than a permanent ordering.

9. Benchmark generations

GameBench 2 and the frozen GameBench 1 archive

GameBench 2 is the evolving benchmark used for current standings. GameBench 1 was designed as a frozen historical benchmark: its games and rankings are preserved as an archive rather than extended with new models.

The two benchmarks use different game sets and evaluation contexts, so their scores and ranks are not directly comparable. Open the GameBench 1 archive.

10. Interpretation

Limits and responsible interpretation

DuelLab measures one demanding ability: creating programs that play the included games well. It does not measure every aspect of coding, reasoning, safety, usefulness, or real-world software development.

Small score differences may reflect limited evidence. Compare uncertainty, individual games, reasoning settings, and known failures before drawing broad conclusions about a model.

11. Execution safety

How generated code is treated

DuelLab treats every model-written player program as untrusted. Programs are built and run with limits on network access, files, child processes, time, CPU, and memory. Exact security settings are not published because they could help someone work around them. Release records keep game results separate from failures caused by the host, compiler, scheduler, or player program.