AI coding models leaderboard

Editorial estimates — not official SWE-bench results

These values are benchr editorial estimates from benchmarks_estimated in models.json. They are shown for planning context only, without a rank or quality-per-dollar calculation.

What SWE-bench Verified measures

SWE-bench presents a model with a GitHub issue and the repository's codebase. The model must locate the bug, write a fix, and have that fix pass the repository's test suite — automatically, without human guidance. The "Verified" subset removes ambiguous issues that could be fixed in multiple ways, making scores more reliable.

What it doesn't measure: speed, latency, non-coding tasks, or how the model behaves when a developer is actively supervising. A model that scores 75% on SWE-bench might be excellent for pair programming even if an 88% model is better for fully autonomous pipelines. The score is most meaningful when humans are out of the loop.

Reading the leaderboard: the cost dimension

Within the provider-published table, the “quality per dollar” column divides a published SWE-bench result by the listed input price. It is a simple derived screen, not a production-cost forecast. Editorial estimates are excluded from both the rank and this calculation.

For autonomous code repair at volume, the relevant question isn't just "who scores highest" but "at what failure rate does the quality gap cost more than the price gap." See the GPT-5 vs Opus 4.8 comparison for that analysis.

Methodology

The ranked table includes only results recorded as provider-published in the data. When no comparable published result is available, any benchr editorial estimate appears in the separate unranked table with a visible label. No private or “internal evaluation” evidence is claimed.

How to validate a coding model before switching

Use SWE-bench as a screen, not as the final decision. A model that repairs Python repository issues well may still struggle with your TypeScript monorepo, your test harness, your dependency graph, or your internal style constraints. Before migration, run the candidate model against recent bugs your team already fixed and check whether its patch would have passed review.

A more useful local metric is accepted patch cost: token spend plus failed attempts and engineer review time. If a cheaper model needs more retries, it can cost more per accepted change. If every patch receives the same developer review, a lower-cost model may still be the better default despite a lower benchmark score.

Why repository fit matters

Coding models vary by language, repository size, test quality, and how much surrounding context they need. A benchmark score can hide those differences. Before moving traffic, sample issues from your own repositories: one simple bug, one dependency problem, one refactor, one failing test with misleading logs, and one issue that requires reading documentation. That small suite reveals failure modes faster than a generic leaderboard.

Also check tool behavior. Some models write excellent patches but struggle with shell commands, file navigation, or concise commit messages. If your coding agent depends on those behaviors, evaluate the full workflow instead of the final patch only.

One more practical point: keep a small regression set after you choose a model. Coding models can change behavior when providers update routing, inference settings, or model aliases. A monthly rerun on the same issues will tell you whether the model still deserves its spot in your pipeline.