Compare two entries
Select two models to inspect their evidence together.
Rankings
Who leads on the chosen metric?
Selected models, ordered by the Rank by choice above; on the leaderboard score the number is each model's official rank
Headline score is the public leaderboard score; the other Rank by choices show diagnostic views when the selected models have that data.
Scores across games
How consistent is each model from game to game?
All settings: each dot is one game (averaged across settings); the ring is the model's headline score, where each failed generation attempt counts as 0
Economics
What does a score cost to generate?
Coming soon
Generation cost versus score will appear when the economics segment is available.
Cost is the generation cost per player program: the equal-weight mean over each model's unique game and reasoning-setting cells, on economics-eligible games only. Each tooltip names its evidence basis (billed, provider-reported, or an estimate from list prices, which is not an invoice) with the median and run count. The score is the headline score on the same games: each failed generation attempt counts as 0, and its cost is included. Every model on this page is a point; the selected models are named. The best-value line runs through the points with complete cost evidence that no other such point matches or beats, that is, none costs the same or less and scores as high, in Per reasoning setting over every model and setting; hollow markers have incomplete cost evidence (known charges only), never join the line and are left out of that comparison. The cost axis is logarithmic, so a cost of $0 sits in its own lane left of the axis break: filled when no charge was recorded on complete evidence, hollow when the evidence is incomplete and the true cost unknown. “All settings” combines the public reasoning settings; Baseline, Balanced and Intensive each select the provider setting assigned to that role for each model, and Per reasoning setting always shows all three. Output-token and API-latency views are under More charts.
Reasoning effect
How much does more reasoning help each model?
Each row shows one model's three settings (headline score, 0–100; each failed generation attempt counts as 0); Gain is Intensive minus Baseline.
Baseline, Balanced and Intensive are each model's own lowest, middle and highest tested provider settings, so the literal setting differs between models; hover a marker or open the numbers to see it. A heavier setting can score lower than a lighter one.
Program reliability
How often does each model write a player program that works?
How each model's generation attempts ended, all reasoning settings pooled
Each bar counts one model's generation attempts in the active reasoning view. Each failed attempt counts as 0 in the headline score, so a model with failures has a headline score below its working-program strength. A program rejected only for being too slow does not play and does not count; attempts that failed for reasons outside the model's code, such as provider, API or test-machine faults, are left out. Working gives the working programs out of the counted attempts.
Working code vs strong code
Does code that works also play well?
Each point is one model on this page; selected models are named
Program success rate is working programs divided by counted attempts, from the same counts as Program reliability: an attempt counts when it produced a working program or failed in the model's own code; a program rejected only for being too slow, and an attempt lost to a provider, API or test-machine fault, does not count. Working-program strength is the score of the programs that worked, with failed attempts left out (the leaderboard's column of the same name); All settings pools the three settings' attempts and averages the strengths of the settings that have one, as the Overall score averages its setting scores. Marker area shows how many counted attempts a rate rests on, so 100% from eight attempts reads as weaker evidence than 100% from 24. A model without a rated working program has no strength: it is listed under the chart, not drawn.
Per game
Where is each model strong or weak?
Headline score per game, averaged across reasoning settings; each failed generation attempt counts as 0
Head-to-head
Who beats whom?
Combined view across tested provider settings.
Each cell is the row model's score against the column opponent: wins plus half the draws, as a share of their matches (W-L-D in the tooltip), so the two cells of a pair add up to 100. Use Reasoning view to show one reasoning setting.
More charts
Less-used views open on demand, so the page stays short.
Game spread Which games separate the selected models most
Each row is one game; dots are the selected models' working-program strength in the active reasoning view (failed generation attempts left out). Games are ordered by the standard deviation of those scores, shown at the right: a wide spread means the game separates stronger from weaker programs, a narrow one that the selected models play it about equally well.
Cycle detector Non-transitive matchups between selected models
What it finds: Leaderboard averages can hide matchup loops. The detector searches the currently selected models and reasoning view for cases where A outperforms B, B outperforms C, and C outperforms A.
How to read it: In A → B → C → A, each arrow points from the model with the higher pair score to the model it outscored. Each result lists its three edges with their pair score and win-loss-draw record.
Each arrow requires at least four matches and a pair score above 50%, with a draw counting as half a win. Results are ordered by the cycle’s weakest edge: weakest edge +x pp is the smallest of the three margins above 50%, while n is the total match count across all three edges. The detector shows up to ten cycles. These are descriptive matchup patterns, not tests of statistical significance. “No cycles” means none met these rules in the current selection and reasoning view; it does not prove that the matchups are transitive.
Output tokens and API latency How much each model wrote, and how fast
These views will appear when the economics segment is available.
Output tokens and API latency are generated-code and repair response telemetry on economics-eligible games; latency excludes local validation and compile probe time. These axes are logarithmic.
Static rankings — current page release
This table remains available without chart scripts. It describes this page's release; restoring a requested comparison and its filters requires JavaScript. Open benchmark rankings for the detailed tables.
Current comparison tables
Overall leaderboard scores
All officially ranked Overall models from the same data as the interactive charts. Partial rows keep their official rank and show that their evidence is incomplete.
| Model | Evidence status | Rank | Mean | Min | Max |
|---|---|---|---|---|---|
 GPT-6 Astra | Complete evidence | 1 | 89.8 | 72.6 | 98.9 |
 Claude Opus 5.5 | Complete evidence | 2 | 88.8 | 52.9 | 99.4 |
| Claude Fable 5.1 | Complete evidence | 3 | 88.5 | 63.8 | 98.2 |
| GPT-6.1 Sol | Complete evidence | 4 | 86.0 | 51.5 | 98.2 |
 Gemini 4 Argon | Complete evidence | 5 | 83.3 | 59.4 | 97.6 |
 Kimi K3.1 Pro | Coverage incomplete: fixed opponent-and-seat coverage is incomplete | 6 | 82.9 | 43.5 | 95.6 |
| Claude Opus 5 | Complete evidence | 7 | 78.2 | 44.8 | 95.9 |
| GPT-6 Sol | Complete evidence | 8 | 74.6 | 10.4 | 94.4 |
| GPT-5.6 Sol | Complete evidence | 9 | 68.4 | 41.4 | 88.7 |
| Claude Fable 5 | Complete evidence | 10 | 67.2 | 20.8 | 93.0 |
 Muse Wonder | Complete evidence | 11 | 66.4 | 17.1 | 93.1 |
 GLM-5.3 | Complete evidence | 12 | 65.5 | 0.0 | 93.0 |
 Grok 4.7 | Complete evidence | 13 | 65.0 | 0.0 | 93.4 |
 DeepSeek V4.1 Flash 1017 | Complete evidence | 14 | 63.4 | 33.3 | 84.3 |
| Claude Sonnet 5.5 | Complete evidence | 15 | 61.7 | 14.3 | 99.0 |
| Grok 4.6 | Complete evidence | 16 | 60.9 | 13.7 | 85.0 |
| Gemini 3.8 Flash | Complete evidence | 17 | 60.7 | 37.1 | 88.6 |
| Muse Spark 1.3 | Complete evidence | 18 | 59.7 | 26.8 | 84.7 |
| Muse Spark Open | Coverage incomplete: some games cannot be scored: their only generation attempts failed because of the test harness or the provider and cannot currently be repeated | 19 | 58.3 | 32.7 | 92.3 |
| Grok 4.5 | Complete evidence | 20 | 58.2 | 15.7 | 90.4 |
| GLM-5.3 FlashX | Complete evidence | 21 | 57.6 | 19.2 | 85.0 |
| Claude Haiku 5.5 | Coverage incomplete: some games cannot be scored: their only generation attempts failed because of the test harness or the provider and cannot currently be repeated | 22 | 57.3 | 16.2 | 91.8 |
| GLM-5.3 Flash | Complete evidence | 23 | 57.0 | 17.2 | 81.4 |
| GPT-5.4 | Complete evidence | 24 | 55.2 | 0.0 | 97.5 |
 Qwen3.8 Max 0902 | Complete evidence | 25 | 55.1 | 26.3 | 92.4 |
| Qwen3.8 2.4T A95B | Complete evidence | 26 | 54.6 | 32.2 | 91.0 |
| DeepSeek V4.1 Flash | Complete evidence | 27 | 53.8 | 24.8 | 82.6 |
 Hy4 Preview | Complete evidence | 28 | 53.6 | 0.0 | 89.4 |
| Muse Spark 1.2 | Complete evidence | 29 | 53.6 | 19.7 | 85.8 |
| Gemini 3.6 Flash | Complete evidence | 30 | 52.5 | 18.8 | 83.9 |
 Ember-1 | Complete evidence | 31 | 52.4 | 19.5 | 86.6 |
 Ox Alpha | Complete evidence | 32 | 52.2 | 20.4 | 74.1 |
| Claude Opus 4.8 | Complete evidence | 33 | 51.8 | 0.0 | 83.3 |
| Gemini 3.5 Flash | Complete evidence | 34 | 51.2 | 11.7 | 90.5 |
| GPT-5.5 | Complete evidence | 35 | 50.6 | 0.0 | 87.0 |
| Qwen3.8 Max | Complete evidence | 36 | 50.0 | 21.1 | 90.6 |
| GPT-6 Luna | Complete evidence | 37 | 49.0 | 27.8 | 86.3 |
| Kimi K3 | Complete evidence | 38 | 48.8 | 0.0 | 93.7 |
| Claude Opus 4.5 | Complete evidence | 39 | 48.7 | 32.3 | 78.2 |
| Claude Sonnet 5 | Complete evidence | 40 | 48.1 | 23.5 | 81.4 |
 Mercury 3 | Coverage incomplete: some games cannot be scored: their only generation attempts failed because of the test harness or the provider and cannot currently be repeated | 41 | 47.8 | 0.0 | 79.6 |
| GPT-5.6 Terra | Complete evidence | 42 | 47.1 | 0.0 | 80.6 |
| Gemini 3.1 Pro Preview | Complete evidence | 43 | 46.5 | 14.9 | 85.7 |
| Beam 1.0 501B A23B | Complete evidence | 44 | 44.7 | 0.0 | 82.7 |
| GLM-5.2 | Complete evidence | 45 | 44.7 | 10.4 | 78.7 |
| DeepSeek V4 Pro 0813 | Complete evidence | 46 | 43.8 | 0.0 | 79.3 |
| DeepSeek V4 Pro | Complete evidence | 47 | 43.8 | 0.0 | 84.6 |
| GPT-5 | Coverage incomplete: one or more documented settings are untested or ineligible | 48 | 43.0 | 0.0 | 75.9 |
| GPT-5.6 Luna | Complete evidence | 49 | 41.6 | 11.3 | 83.6 |
 Minimax M3.1 Flash | Coverage incomplete: the provider capability record is unavailable | 50 | 38.1 | 0.0 | 72.8 |
| o3 | Complete evidence | 51 | 36.2 | 0.0 | 58.7 |
 Seed 2.0 Code | Complete evidence | 52 | 36.2 | 0.0 | 86.9 |
| Qwen3.8 27B | Coverage incomplete: generation quality is below the publication threshold | 53 | 34.4 | 0.0 | 78.2 |
| DeepSeek V4 Flash | Complete evidence | 54 | 34.1 | 0.0 | 82.1 |
| Space Bunny Alpha | Complete evidence | 55 | 33.3 | 0.0 | 74.4 |
| Gemini 3.5 Flash Lite | Complete evidence | 56 | 33.3 | 10.3 | 62.0 |
 Inkling Small | Complete evidence | 57 | 32.0 | 5.5 | 57.1 |
| Muse Glimmer 30B | Complete evidence | 58 | 30.5 | 0.0 | 49.1 |
| GPT-OSS 120B | Complete evidence | 59 | 30.2 | 0.0 | 58.3 |
 Nemotron 3 Ultra 550B A55B | Coverage incomplete: one or more documented settings are untested or ineligible; some games cannot be scored: their only generation attempts failed because of the test harness or the provider and cannot currently be repeated | 60 | 27.5 | 0.0 | 61.9 |
| Inkling | Complete evidence | 61 | 26.7 | 0.0 | 48.9 |
 Step 3.7 Flash | Complete evidence | 62 | 24.2 | 0.0 | 50.7 |
| Mercury 2.5 Preview | Complete evidence | 63 | 21.9 | 0.0 | 46.2 |
| Gemini 3.1 Flash Lite | Coverage incomplete: one or more documented settings are untested or ineligible | 64 | 16.3 | 0.0 | 44.8 |
 Solar Mini 4 | Coverage incomplete: generation quality is below the publication threshold; some games cannot be scored: their only generation attempts failed because of the test harness or the provider and cannot currently be repeated | 65 | 8.2 | 0.0 | 46.1 |
 Granite 4.2 8B | Coverage incomplete: every generation attempt of a reasoning setting failed, so that setting scores 0; generation quality is below the publication threshold | 66 | 1.6 | 0.0 | 38.8 |