Running model comparison
When a notable model comes out, we compare it against the field by running Claude Code with that model on two fixed autoresearch tasks, and we publish the trajectories here: mean and variance over 3+ runs, per evaluation and per wall-clock hour. Why we measure it this way is covered in the blog post.
The latest addition is Opus 5. It has the best Shakespeare mean so far and is second on KernelBench, behind Fable 5.
Current standings
| MODEL | KERNELBENCH @ 0.85 H | KERNELBENCH FINAL | SHAKESPEARE @ 4 H | CADENCE, EVALS/H |
|---|---|---|---|---|
| Fable 5 | 22.0 ± 0.6 | 22.3 ± 0.4 | -1.408 ± 0.045 | 21 |
| Opus 5 | 20.9 ± 0.7 | 20.9 ± 0.8 | -1.356 ± 0.041 | 35 |
| Opus 4.8 | 19.6 ± 0.6 | 19.9 ± 0.3 | -1.395 ± 0.055 | 41 |
| Kimi K3 | 19.5 ± 3.3 | 20.4 ± 4.0 | -1.402 ± 0.012 | 9 |
| Inkling | 13.5 ± 0.0 | 13.5 ± 0.0 | -1.487 ± 0.006 | 13 |
Mean ± std over 3 runs per model per task. "@ 0.85 h" and "@ 4 h" are best scores at fixed cutoffs, the largest budget every run reached on that task; final scores alone would favor models whose runs happened to get more wall-clock. Opus 5 has the best Shakespeare mean, though at n=3 it overlaps Opus 4.8 and Kimi K3. Cadence is measured on KernelBench. Bold is best in column.
Fable 5
The KernelBench leader at the cutoff and at the final. On Shakespeare its mean now trails Opus 5's, within noise at n=3.
Opus 5
The best Shakespeare mean; one run reached -1.28 validation loss, the best single result on this page. Second on KernelBench, about 6% behind Fable 5, at a higher cadence (35 vs 21 evaluations per hour).
Opus 4.8
The highest cadence on the page, 41 evaluations per hour, and about 10% below Fable 5 on KernelBench. Its Shakespeare mean is no longer the best.
Kimi K3
Competitive on average, but the KernelBench mean hides a 15.8 to 22.7 spread across runs. On Shakespeare it is the tightest of the pack (±0.012). Budget extra runs to protect against a stuck one.
Inkling
Well behind on KernelBench (13.5 against Fable 5's 22.3) and last on Shakespeare. Results are task-dependent, so test it on your own problem before ruling it in or out.
Prices are USD per 1M input / output tokens on each provider's first-party API, standard tier, as of 2026-07-26.
Progress curves
Each task gets two views of the same runs: best score so far per evaluation (how well the model spends attempts) and per wall-clock hour (how well it spends your time). Pick the models to compare:
KernelBench
The agent optimizes one fixed LayerNorm problem on an RTX PRO 6000; score is the measured wall-clock speedup over PyTorch eager. The fixed cutoff for the table is at 0.85 h, dashed line.
NanoGPT Shakespeare
The agent tunes the training recipe of a small character-level GPT on Tiny Shakespeare; score is negative validation loss. Every evaluation is a 20 minute training run, so evaluations dominate the wall-clock and the per-evaluation axis is the fair one here. The fixed cutoff for the table is at 4 h, dashed line.
How the benchmark works
- Same harness. Every model runs inside Claude Code (versions 2.1.215–2.1.220 across cohorts) with identical prompts and configs, so the model is the only thing that changes between runs.
- At least 3 runs per model per task. A single run is mostly noise. We report mean ± std and keep every raw trajectory public.
- Fixed budgets. Runs get about an hour of wall-clock on KernelBench and several hours on Shakespeare; the standings table also scores everyone at fixed cutoffs so unequal run lengths can't tilt the comparison.
- New releases get added. When a model comes out, it goes into the queue and the standings update. The task set is growing too.
2026-07-26 · Opus 5 added
Follow the standings
We ping the Discord when a new model is added to this page. Raw trajectories and every individual run: research.autolab.ai.