AI coding models leaderboard

What SWE-bench Verified actually measures

SWE-bench presents a model with a GitHub issue and the repository's codebase. The model must locate the bug, write a fix, and have that fix pass the repository's test suite — automatically, without human guidance. The "Verified" subset removes ambiguous issues that could be fixed in multiple ways, making scores more reliable.

What it doesn't measure: speed, latency, non-coding tasks, or how the model behaves when a developer is actively supervising. A model that scores 75% on SWE-bench might be excellent for pair programming even if an 88% model is better for fully autonomous pipelines. The score is most meaningful when humans are out of the loop.

Reading the leaderboard: the cost dimension

Claude Opus 4.8 leads on raw SWE-bench at 88.6%, but costs $5/1M input. DeepSeek V4-Pro scores 80.6% and costs $0.435/1M. Claude Haiku 4.5 scores 73.3% and costs $1/1M. The table above shows a "quality per dollar" column — SWE-bench score divided by input price — to surface models where the value proposition is strongest. DeepSeek V4-Pro and DeepSeek V4-Flash consistently lead this derived metric.

For autonomous code repair at volume, the relevant question isn't just "who scores highest" but "at what failure rate does the quality gap cost more than the price gap." See the GPT-5 vs Opus 4.8 comparison for that analysis.

Methodology

SWE-bench Verified scores from official provider submissions where published. Where a provider hasn't submitted, benchr uses an editorial estimate based on related benchmarks and disclosed internal evaluations. Estimates are marked with an asterisk in the table. Input prices from official API documentation as of June 3, 2026.

How to validate a coding model before switching

Use SWE-bench as a screen, not as the final decision. A model that repairs Python repository issues well may still struggle with your TypeScript monorepo, your test harness, your dependency graph, or your internal style constraints. Before migration, run the candidate model against recent bugs your team already fixed and check whether its patch would have passed review.

The strongest production metric is accepted patch cost: token spend plus failed attempts plus engineer review time. If a cheaper model needs twice as many retries, the higher-scoring model can be less expensive in practice. If every patch is reviewed by a developer anyway, a mid-tier model may be the better default even when Opus leads the benchmark.

Why repository fit matters

Coding models vary by language, repository size, test quality, and how much surrounding context they need. A benchmark score can hide those differences. Before moving traffic, sample issues from your own repositories: one simple bug, one dependency problem, one refactor, one failing test with misleading logs, and one issue that requires reading documentation. That small suite reveals failure modes faster than a generic leaderboard.

Also check tool behavior. Some models write excellent patches but struggle with shell commands, file navigation, or concise commit messages. If your coding agent depends on those behaviors, evaluate the full workflow instead of the final patch only.

One more practical point: keep a small regression set after you choose a model. Coding models can change behavior when providers update routing, inference settings, or model aliases. A monthly rerun on the same issues will tell you whether the model still deserves its spot in your pipeline.