benchr changelog

Added six official-source model records to model-figures.json: Grok 4.5 from xAI, GLM-5.2 from Z.AI, MiniMax M3, Meta Muse Spark 1.1, Gemini Omni Flash Preview, and Gemini 3.1 Flash Lite Image. The ranked tool layer now includes the three text/API models with official token pricing: Grok 4.5, GLM-5.2, and MiniMax M3. Meta's preview model and Google's media models are tracked in the source-of-truth figures file but excluded from the chat/coding ranking because their pricing or modality does not map cleanly to the ranked table. New pricing pages: Grok 4.5, GLM-5.2, and MiniMax M3.

Gemini 3.5 Pro retracted. The June 30, 2026 claim that Google released Gemini 3.5 Pro was fabricated: both source URLs originally cited now 404, Google's Gemini API pricing and model docs still list Gemini 3.1 Pro as the current flagship, and Google DeepMind's own model hub lists "Gemini 3.5 Pro" as coming soon, not shipped. The review and pricing page are withdrawn and replaced with correction notices at articles/gemini-3-5-pro-review and pricing/gemini-3-5-pro; every other mention across the site has been corrected or removed.

GPT-5.6 availability corrected twice. The July 8 correction correctly walked back benchr's premature July 1 GA claim. OpenAI then confirmed the actual API release on July 9, 2026 in its official changelog, so the current status is GA. Pricing stayed $5/$30, $2.50/$15, and $1/$6; live docs now list 1.05M context and 128K max output for all three tiers.

Claude Sonnet 5 max output corrected 200,000 → 128,000 tokens. Re-read against Anthropic's Models overview docs: Sonnet 5's max output is 128,000 tokens, not 200,000 as recorded at launch. That figure ties Claude Opus 4.8's max output rather than exceeding it, and is double Sonnet 4.6's 128,000 rather than triple. Corrected in model-figures.json, models.json, the Sonnet 5 launch and pricing pages, and every comparison that cited the old figure.

New deprecation: DeepSeek's legacy deepseek-chat and deepseek-reasoner aliases. Per api-docs.deepseek.com, these legacy aliases retire July 24, 2026 in favor of the explicit versioned names (DeepSeek V4-Flash / V4-Pro) — an alias rename only, no pricing or capability change. Added to deprecations.json, the deprecations hub, and the tracker.

Full writeups of the three corrections above are logged at /corrections.

Corrected Claude Sonnet 5 pricing in model-figures.json, models.json, history.json, and the English/Arabic Sonnet 5 launch and pricing pages. Anthropic's official rate is $2/1M input and $10/1M output through August 31, 2026, then $3/$15 from September 1. The July 1 benchr entry previously recorded $4/$20 incorrectly.

OpenAI's GPT-5.6 system card was updated on July 1, 2026 with new benchmark scores and confirmed specs while access was still limited to preview partners. The July 9 API changelog later confirmed GA. Sol is benchr's canonical models.json entry: $5/$30 per 1M, cached input $0.50, 1.05M context, 128K max output, API id gpt-5.6-sol. Terra and Luna are tracked in model-figures.json with official prices $2.50/$15 and $1/$6 and the same 1.05M context / 128K output limits.

Correction, July 8, 2026: this entry originally added a "Gemini 3.5 Pro" model that Google never released — the claim was fabricated, and the addition has been retracted. See the July 8, 2026 entry above and /corrections for the full explanation.

Claude Sonnet 5 (released July 1, 2026) — Anthropic's second Mythos-class-architecture model after Claude Fable 5 — added to models.json (id claude-sonnet-5): 1,000,000 context, 128,000 max output (matching Claude Opus 4.8's, and double Sonnet 4.6's 128,000), tentative retirement floor not sooner than July 1, 2027. SWE-bench Verified 89.4% edges out Claude Opus 4.8's 88.6%, though Opus 4.8 still leads deep reasoning with GPQA Diamond 93.6% vs Sonnet 5's 92.0%. Terminal-Bench 2.1 85.6%. Pricing corrected July 3: $2/1M input and $10 output through August 31, then $3/$15 from September 1; the earlier $4/$20 entry was wrong. New launch writeup and pricing page.

The export-control review that suspended Claude Fable 5 for all customers on June 12 closed July 1, 2026: the Commerce Department lifted the suspension and AWS restored Bedrock access in step, the same day Anthropic launched Claude Sonnet 5. Claude Mythos 5 remains Project-Glasswing-only as before — unaffected by this restoration. No pricing or spec changes: $10/$50 per 1M, 1M context, 128K max output are unchanged. Recorded as a dated availability note on the Fable 5 entry in model-figures.json; updated launch writeup.

On June 26, 2026 OpenAI began a limited preview of three GPT-5.6 models: Sol (flagship), Terra (GPT-5.5-class at roughly half the price), and Luna (cheapest and fastest), plus new max and ultra reasoning modes. Access is restricted to about 20 US-government-approved trusted partners via the API and Codex, with general availability planned “in the coming weeks.” Announced pricing per 1M tokens is Sol $5/$30, Terra $2.50/$15, Luna $1/$6 — recorded in model-figures.json with a clear caveat: those rates come from OpenAI's own preview announcement and help center, but as of June 28 they are not yet on the machine-readable developers.openai.com pricing page (re-read live — it still lists 5.5/5.4), and context, max output, and the exact API IDs are unpublished. New GPT-5.6 launch writeup and pricing page. Held out of the interactive ranking tool for now — no official benchmarks to score it on, the same bar we applied to Kimi K2.7-Code.

The June 12 U.S. export-control suspension of Anthropic's Mythos-class models eased on June 26: the Commerce Department cleared Claude Mythos 5 for redeployment to a defined set of US critical-infrastructure organizations — a limited Project Glasswing re-release, not general availability. Claude Fable 5, the generally available model, remains fully suspended for all customers; the clearance did not cover it, and its $10/$50 listed pricing is unchanged. The same June 2 “covered frontier model” Executive Order is the backdrop for OpenAI's government-gated GPT-5.6 release. Recorded on the Fable 5 and Mythos 5 entries in model-figures.json; background on the export-control explainer.

Both image-preview models reached their June 25, 2026 shutdown — re-confirmed on Google's official Gemini deprecations table. They were flipped deprecated → retired in deprecations.json (the hub moves them into the retired record); migrate to the GA gemini-3-pro-image and gemini-3.1-flash-image. Still upcoming: Claude Mythos Preview retires June 30, 2026.

Added to model-figures.json: Mistral OCR 4 (per-page $4/$2/$5 per 1,000 pages), Mistral Small 4 ($0.15/$0.60, 119B/6B MoE, 256K), Qwen3.7-Max (hosted, $2.50/$7.50), Grok Build 0.1 (agentic coding, $1.00/$2.00, 256K), and the open-weight Qwen-AgentWorld 35B-A3B and 397B-A17B. Corrections from the same pass: Qwen3.7-Plus and Qwen3.6-Plus now carry official per-token prices on Alibaba Cloud Model Studio (resolving the prior “not published” null) — tiered $0.4–$1.2 / $1.6–$4.8 and $0.5–$2 / $3–$6 respectively; and Grok 4.20 now lists $1.25/$2.50 at 1M context on docs.x.ai, reconciled from the $2/$6, 2M launch figure. No commercial price changed at any other tracked provider since June 23.

Moonshot's coding-specialist successor to K2.6 is now in model-figures.json: Kimi K2.7-Code at $0.95/1M input (cache miss), $0.19 cache hit, $4.00 output, 256K context, Modified MIT — plus a Highspeed tier at double the rates ($1.90/$0.38/$8.00) for ~180 tok/s. Both verified June 23 on platform.kimi.ai and the official Hugging Face model card; Moonshot publishes no official release date. New Kimi K2.7-Code pricing page, and the Kimi review now carries an update section. Held out of the ranking tools for now — no official benchmarks to score it on.

A full re-check against provider pages found no commercial price change at any tracked provider since June 12. GPT-5.5 Pro standard pricing is now readable on developers.openai.com — $30/1M input, $180/1M output (batch $15/$90, no cached-input discount) — so the previously-null fields in model-figures.json are filled, and the GPT-5.5 pricing page notes the tier. Corrections from the same pass: Mistral Medium 3.5's official release date is April 28, 2026; the hosted Qwen flagships Qwen3.6-Plus (1M context), Qwen3.6-Max-Preview, and Qwen3.7-Plus (multimodal) are confirmed on Alibaba Cloud — per-token pricing isn't officially published, so it stays null.

Logged OpenAI's June 11, 2026 notice: the dated snapshots gpt-5-2025-08-07, gpt-5-mini/nano-2025-08-07, gpt-5-pro-2025-10-06, o3-2025-04-16, and o3-pro-2025-06-10 are removed from the API December 11, 2026. The floating gpt-5 / gpt-5-mini / gpt-5-nano aliases stay live. Added to deprecations.json, the deprecations hub, and the tracker. Imminent and unchanged: Gemini 3 Pro Image Preview (June 25), Claude Mythos Preview (June 30).

Their June 15, 2026 shutdown has passed — re-confirmed on Anthropic's official model-deprecations table. Calls to claude-sonnet-4-20250514 and claude-opus-4-20250514 now fail. Both were flipped deprecated → retired in deprecations.json; the deprecations hub moved them into the retired record, and the Sonnet 4 and Opus 4 migration pages now read as in-effect. Opus 4.1 remains deprecated until August 5, 2026.

On June 12 a U.S. Commerce Department export-control directive barred Mythos-class models from use by any foreign national, and Anthropic suspended both Claude Fable 5 and Claude Mythos 5 for all customers; AWS revoked Bedrock access the same day. All other Claude models are unaffected. Listed pricing and specs are unchanged in model-figures.json (a dated availability note was added); Anthropic disputes the recall and says it is working to restore access. Coverage updated on Claude Mythos, the Fable 5 launch, and the Fable 5 pricing, comparison, and model-picker pages.

New /errors reference: 15 common AI API errors across OpenAI, Anthropic, and Gemini, each verified against the provider's official error documentation, with causes, code fixes, and migration bridges into the deprecations record and pricing pages. Structured dataset at api-errors.json; validator at scripts/generate_error_pages.py.

Every tracked commercial price re-read live against the provider's own pricing page — no price changes since June 10 (Anthropic, OpenAI, Google, xAI, DeepSeek, Moonshot, Mistral). Anthropic's pricing page now publishes per-model cache-hit prices, closing previously-honest gaps in model-figures.json: Opus 4.8/4.7 $0.50, Sonnet 4.6 $0.30, Haiku 4.5 $0.10, Fable 5/Mythos 5 $1 per 1M cached input.

Status correction: Claude Opus 4.6 is officially Active (the entry previously called it legacy), with fast mode at $30/$150. Anthropic's official "not sooner than" tentative retirement floors logged for all active Claude models (e.g. Opus 4.8 → May 28, 2027; Sonnet 4.6 → February 17, 2027).

Deprecation record expanded by 9 entries from a full re-read of provider docs: OpenAI's July 23 wave (five Codex IDs, o4-mini-deep-research, computer-use-preview), GPT-4.1 nano and GPT Image 1 joining the October 23 list (now eleven IDs), the November 30 platform retirements (Prompts API, Evals, Agent Builder — announced June 3), and Gemini 3 Pro Image Preview (June 25). Details on the deprecations hub. Gemini 3.5 Pro remains unreleased — nothing official to add.

New /deprecations hub: every announced retirement across Anthropic, OpenAI, and Google with shutdown dates, replacements, and migration price math — plus five dedicated migration pages (Claude Sonnet 4, Claude Opus 4/4.1, GPT-4o, OpenAI's October wave, Gemini 2.5), a deprecations RSS feed, and an open deprecations.json dataset. All entries verified June 12 against official provider deprecation docs.

New /price-history page: the append-only record of verified AI API pricing events, published as open data (CC BY 4.0) from history.json.

Tracker corrections from the same verification pass: GPT-4o and GPT-4 Turbo retire October 23, 2026 (the tracker previously showed early-2026 dates); Gemini 3 Pro Preview's API shutdown was March 9, 2026 (not March 26); Claude 3 Opus (Jan 5) and Claude 3.5 Haiku (Feb 19) are now marked retired; Claude Mythos Preview (June 30) and Gemini 2.5 Pro/Flash (October 16) added.

Claude Fable 5 (released June 9, 2026) added to the index: $10/$50 per 1M, 1M context, 128K output, API id claude-fable-5. First Mythos-class model in general availability. Source: anthropic.com/news/claude-fable-5-mythos-5.

GPT-5.4 re-added. It was removed on June 1 as "unverified" — that call was wrong. Re-verified June 10 against OpenAI's pricing page and release notes: released March 5, 2026, $2.50/$15 per 1M, cached input $0.25, context up to 1M. The same pass confirmed Claude Opus 4.6 (Feb 5), Grok 4.20 (Mar 10, $2/$6, 2M context), and Qwen 3.5 (Feb 16) are all real; entries added to model-figures.json.

Tracker updated: Claude Sonnet 4 and Claude Opus 4 retire June 15, 2026; Claude Opus 4.1 retires August 5, 2026 (per the official Anthropic docs).

Launched the full benchr tool suite: ranked index, charts, cost calculator, model recommender, deprecation tracker, and release timeline. All tools read from models.json — never duplicate data per page.

New models added to the index: Claude Opus 4.8, GPT-5, GPT-5 Mini, Grok 4.3, Kimi K2.6, Mistral Large 3, Llama 4 Scout, Llama 4 Maverick. Removed unverified entries (GPT-5.4, GPT-5.4 mini, Qwen 3.5). All 19 current models verified against official provider pricing pages.