Small language models, in working use

A support-email router is a useful example because it has a finite label set and can be evaluated against human decisions. This page previously described an unpublished support-email comparison as a benchr result without publishing the dataset or protocol; that claim has been withdrawn. The defensible approach is to sample your own inbox, remove sensitive data, create held-out labels, and compare candidates before deployment.

If your workload involves classification, extraction, or routing with a tight latency budget and a recoverable failure mode, Phi-4 mini and Gemma 3 4B deserve a careful look. That is a candidate-selection recommendation, not a claim that every small model reaches a fixed percentage of frontier quality.

This piece covers the sub-10B-parameter tier and a reproducible decision framework. These models are worth evaluating when privacy, hardware control, or latency dominates your constraints. Once your workload depends on multi-step reasoning, long-context synthesis, or broad world knowledge, the frontier tier is usually the safer candidate.

(A side note before the category boundaries: Mistral 7B remains relevant to teams with an established self-hosted deployment. This page narrows its current candidate set to Phi-4 mini, Gemma 3 4B, and Qwen3 8B; that scope is not evidence that an existing Mistral deployment should be replaced.)

What "small" means here

Anything under 10B parameters. Microsoft documents Phi-4 at 14B and Phi-4 mini at 3.8B. Google's Gemma 3 model card lists 1B, 4B, 12B, and 27B variants. The official Qwen3 release lists dense models at 0.6B, 1.7B, 4B, 8B, 14B, and 32B. This guide focuses editorially on the 4B-to-9B band. Some four-bit artifacts can fit within 16GB, but weights, KV cache, context length, runtime overhead, and other processes all consume memory. Verify the exact artifact on the target device; see running models on your own machine.

The parameter counts and training costs of closed frontier models are generally not disclosed, so this page does not invent a size ratio. The practical comparison is narrower: can an open-weight candidate clear your task-quality threshold while improving the latency, privacy, or total-cost constraints that matter to your deployment?

The provider documentation establishes multilingual scope, not a universal ranking: Gemma 3 documents support for more than 140 languages, while the Phi-4 mini card lists 24 supported languages. Neither source proves which model is better for your language, register, or structured-extraction schema. Include representative held-out examples and fluent review, especially for Arabic and mixed-language text.

Three workloads where small models may fit

Classification and extraction. Routing, triage, and structured extraction are bounded enough to evaluate with labeled examples. A small model is attractive only when its measured error rate and total operating cost fit the workload; do not assume a fixed quality or cost ratio.

Routing and triage. The model decides where your request should go: which API to call, which downstream model to invoke, or which template to apply. A small model can work as a router when the label set is bounded and errors are recoverable. Validate it against human labels and define the confidence or review rule that escalates uncertain requests.

On-device or private inference. When data cannot leave an approved environment, a locally deployed model such as Gemma 3 4B may be a candidate if its quality and security controls are acceptable. Local execution is not by itself proof of regulatory compliance; architecture, logging, access, updates, and human review still matter.

Where to begin evaluating small models

Editorial workload-fit guide, not measured accuracy. Validate every candidate on your own held-out examples.

Where to be cautious

Multi-step reasoning. A compact model may lose coherence across a long chain even when it can perform individual steps. Do not assume a fixed failure jump or that a larger model always succeeds; evaluate complete tasks, intermediate constraints, and consequential failure cases.

World knowledge. A compact model can produce fluent but unsupported answers on obscure or changing facts. Use retrieval from authoritative sources, citations, abstention rules, and human review where factual error matters; parameter count alone does not establish reliability.

Long-context retrieval. An advertised context window does not establish accurate retrieval or reasoning near its limit. Test answerable positions, distractors, citations, and failure handling across the document lengths you use. The context-window piece covers that distinction in detail.

Treat 4B–9B as a candidate tier, not a quality guarantee. The workload decides where it belongs.

Candidates worth including

Microsoft's Phi-4 mini model card documents a 3.8B-parameter model released in February 2025, a 128K context window, 24 supported languages, and an MIT license. Those facts make it a relevant compact candidate; they do not prove that it is the strongest model for a particular structured workload.

The larger Phi-4 at 14B sits outside this page's sub-10B scope. It can still serve as a higher-capacity local comparison where its artifact fits, using the same prompts, rubric, and operating-cost calculation.

Google's Gemma 3 model card documents the 4B variant, a March 2025 generation with text-and-image input, a 128K context window, and support for more than 140 languages under the Gemma Terms. That documented breadth makes it relevant to a multilingual candidate set, but the source does not establish Arabic quality or superiority over Phi-4 mini or Qwen3 for your workload.

Documented open-weight candidates, reviewed July 2026

Model	Params	License	Evaluation role
Phi-4 mini	3.8B	MIT	Compact structured-task candidate
Phi-4	14B	MIT	Higher-capacity local comparison
Gemma 3 4B	4B	Gemma Terms	Multilingual and image-input candidate
Qwen3 8B	8B	Apache 2.0	Multilingual and coding candidate
Llama 3.1 8B	8B	Llama 3.1 Community	Ecosystem baseline

Small-model accuracy is workload-specific. A narrow inbox taxonomy and free-form sentiment analysis test different capabilities, so a result from one cannot be transferred to the other. Publish your label definitions, held-out split, model version, prompt, and review rule internally so the decision remains reproducible.

A concrete production scenario

A concrete evaluation plan makes the trade less abstract. Suppose an inbox-classification pipeline currently uses a frontier API and you are considering Phi-4 mini locally. Define the priority categories, sample and label representative messages, and keep a held-out set neither prompt design nor fine-tuning can see.

Before-and-after numbers:

Evaluation worksheet: frontier API versus a local small model

Metric	Sonnet via API	Phi-4 mini local
Cost per email	Calculate from cited token rates and actual tokens	Allocate hardware, power, storage, review, and operations
End-to-end latency	Measure from your region under realistic load	Measure on the target device and concurrency
Accuracy vs. human labels	Score on the same held-out set	Score on the same held-out set
Data path	Sent to an external API under its terms and controls	Can remain on approved infrastructure if configured that way

The outcome is not known in advance. The local option wins only if it clears your quality threshold and its total cost, latency, privacy, and operating burden are better for this workload.

For a sales-lead routing system where a misclassification has dollar consequences, the trade would tip the other way and the frontier API would stay. Small models open a different operating point on the cost-accuracy curve. The right question isn't which model is better. It's which operating point fits the workload. For the broader pricing picture across workloads, see price per use case.

Fine-tuning may change the result, but that is another experiment with its own held-out set and maintenance cost. Multimodal variants are not evaluated on this page. Both remain questions for your workload-specific assessment.

Small models can occupy useful production roles when a measured quality threshold, privacy design, and total operating cost support that choice.

Classification, extraction, routing, and on-device inference are the workloads most worth screening for a smaller model. At serious volume, even moving a subset of calls can matter—but the share must come from your request logs and evaluation results, not a generic percentage.

A two-tier architecture is one option at volume: use a compact model for bounded requests that clear your threshold, then escalate the rest. It is not universally right; routing errors, duplicated infrastructure, review, and maintenance can outweigh token savings.

For structured work, include Phi-4 mini in the evaluation. For multilingual or image-input work, include Gemma 3 4B; Qwen3 8B is another documented multilingual candidate. None is the default winner: use the same held-out set, exact artifacts, and total-cost calculation.