Context windows compared, across four frontier models

Most context-window benchmarks measure the wrong thing. They report how much text fits, and stay silent on what the model can actually locate once it's in there.

1M tokens at Claude, per Anthropic's API documentation. 1M at Gemini 3.1 Pro Preview, per Google's Gemini models page. 1M at GPT-5, per OpenAI's platform docs. The marketing pitch behind these numbers has been that more context equals more capability, and that long-window models will simply retire retrieval as a relic of the small-window era. What you find in practice is messier than the pitch. The long window earns its keep in a narrow set of workflows, sits there as expensive overhead in most others, and loses outright to proper retrieval in a third category teams keep trying to force into it.

This piece offers a reproducible comparison plan for four long-context implementations; it does not claim that benchr ran them on the same private workload. Per-token costs are drawn from Anthropic's pricing page, OpenAI's API pricing, and Google's published rates. Llama 4 weights and license terms are documented at llama.com. Whether long context or retrieval is better depends on the question, corpus, cache behavior, quality threshold, and measured token use. For the case against using long context as a default, see the million-token marketing piece.

Advertised numbers versus working numbers

Advertised limits, prices, and editorial stress-test starting points; the latter are hypotheses, not measured results

Model	Advertised context	Editorial stress-test starting point	Cost per 1M input tokens
Claude Opus 4.7	1M	Probe around ~600K, then vary length and task	$5
Gemini 3.1 Pro Preview	1M	Probe around ~800K, then vary length and task	$2
GPT-5	400K	Probe around ~250K, then vary length and task	$1.25
Llama 4 Maverick	1M	Probe around ~250K, then vary length and task	varies (self-host)

The third column is an editorial test plan derived from heterogeneous public long-context literature, not a consensus estimate or a comparable head-to-head result. The numbers tell you where to add stress-test cases, not where reliability is proven to stop. Evaluate single-fact lookup, multi-fact synthesis, distractors, ordering, citations, and omissions at several lengths; the usable boundary depends on the corpus and rubric.

No cited controlled benchmark establishes a universal winner or a fixed degradation point across these four models. Use the advertised limits to construct comparable cases, and use the third-column figures only as extra probe points. The result you publish should come from your held-out corpus and scoring rubric.

Needle-in-haystack and multi-fact synthesis tests measure different behaviors, and results from different papers are not automatically comparable. Run both test families with the same model versions and inputs; this page does not convert heterogeneous reports into a measured reliability zone.

Three workload shapes, three different verdicts

To make the pattern concrete, picture a 280,000-token government policy report (about 200 pages of dense prose) and three different questions you might ask of it. The three workload shapes that follow turn up in legal review and research synthesis, and in any cross-document analysis you might run.

Workload one: the broad pillar question. What are the three pillars of the document and what does it say about progress on each? This is an evaluation design, not a result. Score whether each candidate identifies all three pillars, cites the relevant sections, preserves the requested depth, and flags uncertainty; the cited sources do not establish a winner for this hypothetical report.

Workload two: the precise lookup. What is the specific metric the report uses for private-sector contribution to GDP, and what are the current and target values? Compare full-context and retrieval pipelines on exact-match accuracy, source citation, latency, and total query cost. Do not assume either will return the right answer; retrieval quality depends on chunking and ranking, while long-context quality depends on placement, distractors, and model behavior. The RAG vs fine-tuning piece has a cost framework.

Workload three: the cross-section synthesis. Are there internal inconsistencies between the housing-affordability claims in the early chapters and the GDP-mix projections in the later chapters? Test whether each approach finds both sections, explains the relationship, and cites the evidence. A basic top-k retrieval setup may miss a distant dependency, while query expansion, graph retrieval, or a long-context pass may recover it; the page does not assert a winner.

The architectural choice is empirical. Use these three workload shapes to learn where long context, retrieval, or a hybrid clears your quality and cost thresholds.

Long context is worth testing for cross-section questions; retrieval is worth testing for precise lookup. Measure both before choosing the production path.

1. 2022 4K · GPT-3.5

   One letter, one email, one short article. That was it.

2. 2023 32K · GPT-4 Turbo

   A short report, a small codebase, a long memo.

3. 2024 200K · Claude 2

   A novella, a long technical document, a production codebases.

4. Feb 2024 1M · Gemini 1.5 Pro

   First mainstream million-token context. A textbook in one prompt.

5. Sep 2025 10M · Llama 4 Scout

   A 10M advertised limit; usable quality still requires local testing.

The probe-point numbers are editorial starting hypotheses assembled from heterogeneous public material, not measured reliability zones. They do not generalize automatically to legal, scientific, policy, code, structured-data, or conversational corpora. Record the model version, input construction, question set, scoring rule, latency, and cost when you test them.

The cost picture

The 280,000-token version of the query, on Claude Opus 4.7, costs about $1.40 per question in input tokens. The same question answered against a proper vector store with the relevant chunks retrieved costs around $0.04. That's a 35× difference. At one question a day, nobody notices. At 500 questions a day, that gap decides your architecture for you.

Caching changes this picture a lot. If the same long document gets queried repeatedly, Claude's prompt cache drops the input cost on later queries to roughly 10% of the standard rate. Gemini's caching works on the same mechanism and comes out cheaper in absolute terms. With caching on, the long-context query against a frequently-reused document costs roughly $0.40 per question on Claude. That is still ten times what retrieval would charge, but it lands inside the range where the workflows that need long context can justify paying it. For the broader cost picture across workloads, see price per use case.

The decision rule

For exploratory or cross-section questions, include long context in the evaluation because it can expose distant parts of a source to one model call. Also test a retrieval or hybrid baseline; query expansion and multi-stage retrieval can sometimes surface the same evidence at lower token cost.

For precise lookup, retrieval is a strong starting architecture because it can reduce the input passed to the model. Verify ranking recall, answer accuracy, latency, and end-to-end cost rather than assuming it wins every dimension.

For high-volume question answering against a fixed corpus, compare retrieval first and price a cached long-context or hybrid alternative. The sensible choice depends on cache-hit rate, corpus updates, retrieval operations, failure cost, and measured quality.

For corpora that exceed a model's context limit, the system must select, compress, retrieve, or split material before or across calls. Which mechanism works best remains a system-design question.

Million-token limits create new evaluation options, but advertised capacity alone does not demonstrate usable quality for a workload. Treat long context and retrieval as candidate components, and choose from reproducible quality and cost measurements.

No cited controlled head-to-head establishes a strongest long-context model among these four. Include candidates that meet your advertised-capacity, modality, deployment, and price constraints, then compare them on the same held-out corpus. The ~600K, ~800K, and ~250K points above are places to probe, not rankings or promised boundaries.

In a production system, a retrieval baseline is often useful because it makes selected evidence and token use visible. Add long-context and hybrid candidates where cross-section questions justify them, then route by measured quality and cost. For a broader evaluation framework, see RAG vs fine-tuning, with the math.