benchr
What this guide covers
This guide covers the open-weight tier in 2026: the frontier-class open models (Llama 4, Mistral Large 3, DeepSeek-V4, Qwen3.6), the small-model tier (Phi-4 mini, Gemma 3, the new Phi variants), and the hardware question — what does it take to run any of these yourself, and when is that the right call instead of paying for an API.
The frontier-open tier
-
The open-weight tier right now: Llama 4, Mistral, Qwen, DeepSeek
A provider-sourced candidate map after the April 2026 releases. Compare license terms, model-card results, code and multilingual workloads, memory requirements, and deployment cost without assuming a fixed gap to closed models.
The small-model tier
-
Small language models, in working use
Phi-4 mini, Gemma 3, and a reproducible framework for deciding whether a sub-10B model fits a bounded workload. Build a held-out set and report accuracy, review rate, latency, memory, and cost; no private score is implied.
Running them yourself
-
Running models on your own machine
A local measurement plan across runtimes and quantizations. Record the exact checkpoint, quantization, runtime, chip, context length, prompt, tokens per second, power, and utilization before comparing local deployment with an API.
-
Local AI reference
Inspect source-linked local model records, keep total and active parameters separate, and plan visible Q4 weight headroom before choosing an exact artifact.
Which open model should you use?
For a permissively licensed production candidate, consider Qwen3.6-27B. Qwen's official card identifies an Apache 2.0 license and reports 77.2% on SWE-bench Verified; that is a provider-published result, not an independent benchr reproduction. Check the exact artifact, quantization, VRAM headroom, and multilingual quality locally. The Local AI planner makes the initial weight-headroom assumptions visible before you test.
If code or math is central and the documented MIT terms fit your use case, include DeepSeek-V4. Read its benchmark figures as DeepSeek-published results, compare it with Qwen on held-out tasks under the same harness, and verify the live pricing page before budgeting; neither benchmark placement nor lowest price is universal.
For classification, extraction, or routing, include Phi-4 mini as a small-model candidate after confirming the official artifact and license. Use a held-out dataset, a fixed prompt and label policy, and your target device to measure accuracy, abstentions, latency, memory, and review cost; this guide does not publish a private email score or laptop throughput result.
For the open-vs-closed cost question, see the AI costs guide. Compare dated provider prices with your own GPU utilization, operations time, retries, review burden, and held-out quality. There is no fixed capability or price gap: it changes with the workload, serving configuration, volume, and evaluation rule.