Running models on your own machine

Running large language models locally has gone from hobbyist project to legitimate production choice in the past eighteen months. The public writing about it has become unreliable in the way these things always do. Every guide oversells the experience. Every benchmark uses cherry-picked configurations. The comparison numbers rarely include the operational cost of running your own infrastructure. This piece is the corrective. Real numbers on a representative machine, an account of what works, and the conditions under which local inference actually pays off against the API.

The example rig used throughout is a current-generation Apple Silicon workstation with 64GB of unified memory. The kind of machine that costs about $3,200 and is accessible to a serious independent developer or a small team. The numbers will scale predictably to larger and smaller machines on the same architecture. For NVIDIA workstations or rack-mounted gear, the numbers differ, but the operational lessons hold. If you want to pick the actual model to run, see the open-weight tier and small language models.

Worth flagging: this piece tests on Apple Silicon. NVIDIA workstations have different software (vLLM, TensorRT-LLM, exllama) and different throughput characteristics. The operational lessons hold across both — the specific tokens-per-second numbers don't. If you're evaluating an NVIDIA setup, treat this as orientation, not prescription.

The software worth using

Four runtimes were tested over six months. Two are worth keeping.

Ollama is the runtime to recommend to anyone starting today. A small Go binary that runs as a background service, manages model downloads via a registry-style command, and exposes a clean HTTP API on port 11434. The defaults are sensible. The model library is broad and current. Installation is one command. From cold start to talking with a local model is under five minutes for a competent user.

llama.cpp is the runtime to use for production. A C++ project with hand-optimized kernels for Apple Silicon, NVIDIA, AMD, and CPU. It's way faster than Ollama on the same model in some configurations, and it exposes parameters Ollama hides. The cost is that it needs manual compilation, manual model file management, and more documentation reading than is strictly fun. Ollama is built on top of llama.cpp, so this isn't a rejection of Ollama so much as a different layer of the same stack.

LM Studio wraps llama.cpp in a desktop GUI. The GUI is good for browsing models and comparing them side by side. benchr didn't adopt it because server-style deployment was preferred and the permanent dock presence was distracting. For GUI-first users, this is the friendliest entry point.

MLX is Apple's first-party machine learning framework with full Apple Silicon support. It produces the fastest tokens-per-second numbers on some models on the test rig. The ecosystem is younger. Fewer models, fewer integrations, more rough edges. For an Apple-Silicon-only deployment that needs the absolute fastest inference, MLX is worth the time. For a mixed stack, it isn't.

Actual numbers, three quantizations

The benchmarks below were collected on the example rig under Ollama with default settings, generating 500 tokens of output from a 200-token prompt, averaged across five runs. Context length set to 8K. Reported values are tokens per second on the output stream.

Local inference on a 64GB M3 Max, Ollama default settings, January 2026

Model	Quantization	RAM used	Tokens/sec	Quality vs. fp16
Llama 4 Scout 88B	Q4_K_M	52 GB	18.2	Clear drop
Llama 4 Scout 88B	Q5_K_M	60 GB	15.7	Subtle drop
Llama 4 Scout 88B	Q6_K	61 GB (tight)	14.1	Basically none
Phi-4 mini 3.8B	Q4_K_M	2.6 GB	110.5	Clear drop
Phi-4 mini 3.8B	Q5_K_M	3.1 GB	98.2	Subtle drop
Phi-4 mini 3.8B	Q8_0	4.6 GB	83.1	None

A few notes on what those numbers mean. The Llama 4 Scout 88B Q6_K configuration runs tight on a 64GB machine. Most other applications need to be closed, and serious foreground work on the same machine becomes impractical while inference is running. The Q4_K_M variant is way more comfortable and the quality drop is real but not crippling for casual chat use. For production tasks where every token matters, Phi-4 mini at Q5_K_M is the configuration to default to. 98 tokens per second with almost no measurable quality cost.

For reference, Claude Opus 4.7 through the API streams at roughly 70-80 tokens per second on a decent connection. The local Phi-4 mini is faster on raw throughput. It's just running a much smaller model.

Local inference doesn't beat the frontier API on capability. It wins on the things APIs can't give you: privacy, control, and a flat per-token cost of zero.

One genuine uncertainty: the break-even point with the API depends heavily on workload shape, electricity rates, and how aggressively you use the hardware off-hours. The $80-100/month threshold I cite is a rough estimate from my own usage and three teams I've talked to. Your number could be half that or double, depending on how you actually use the machine.

The cost-benefit, written down without selling anything

Hardware amortized over four years works out to about $67 per month. Electricity on industrial rates costs around $4 per month for sustained inference on this hardware. Total monthly baseline: $71 before a single token is inferred.

For comparison, a typical small-team API workload (call it $80 to $140 per month) puts the API and the local hardware at roughly break-even on direct cost. That math alone isn't the case for buying the hardware. The case is that the hardware buys you three things the API can't. For the full API-cost picture, see price per use case.

• Privacy on the data the model processes. Sensitive material (customer records, internal documents, anything covered by a residency rule) never leaves your network.
• Latency that's basically zero on the local network. First-token latency on a frontier API from a typical residential connection is 600 to 1,100 ms. From the local rig it's around 80 ms. For interactive workflows that pile up across many turns, the difference is the difference between fluid and laggy.
• Flat marginal cost regardless of volume. The model can be hammered with bulk classifications, fine-tuning experiments, batch processing. None of it costs a cent more per request. That changes what's affordable to try, which in turn changes what gets tried.

If those three properties are valuable to your work, the local setup pays for itself even at direct-cost break-even. If they aren't, the API is just easier.

One honest admission: I haven't tested local models under sustained 24/7 load. The numbers reported here are from interactive use over weeks, not from a production workload running thousands of requests per hour. The fan-noise problem and thermal-throttling problem at sustained load are both real and not in scope.

Where local actually pays off

Three workloads where local is the right pick in 2026, named specifically.

Structured extraction from inbound documents. Support emails, contract drafts, application forms. Run them through Phi-4 mini at Q5_K_M and get structured JSON back. Cost is zero per document, latency is acceptable, and the data stays put. Accuracy is two percentage points below what the Sonnet API would produce on the same input, which is a fair trade for this workload.

Bulk content rewriting against a fixed corpus. Hundreds of feature blurbs, product descriptions, microcopy passes. Anything where running every iteration through an API would add up to real money. The local model produces drafts. You edit them. The cost is the time you're already spending.

Speculative experimentation an API budget would discourage. Generating 5,000 synthetic examples to train a classifier costs about $25 on the API and $0 locally. The first number causes the experiment to not happen. The second causes it to happen, and sometimes the experiment works.

Where local isn't the right answer

Anything that needs frontier capability. The local models, even the largest ones that fit on a 64GB machine, stay clearly behind Claude Opus 4.7 and GPT-5 on hard reasoning, multi-file code understanding, and the kind of voice-sensitive writing where the model's tone matters. Trying to use a local 88B model for the work that should go to the frontier produces a long, painful fight against the gap.

Anything heavily multimodal. The local image-understanding story is way weaker than the closed APIs. Gemini 3.1 Pro Preview through its API is the model for vision tasks. Local isn't competitive. See the multimodal ranking for the full picture.

Anything where your team has no appetite to maintain the setup. Local means handling updates, debugging memory pressure, reading the changelog when llama.cpp ships a breaking change. If your team doesn't want to be its own ops team, the API is correct and the local setup is a distraction.

A reference setup

A working local-inference stack on this kind of machine looks like this. Ollama running as the foreground runtime for ad-hoc work. llama.cpp compiled from source, behind a small Go HTTP server handling authentication and queue management, for production inference. Three models loaded into rotation at any time: Phi-4 mini Q5_K_M for high-volume work (model card on Hugging Face), Llama 4 Scout 88B Q4_K_M for hard tasks, and Qwen 3 7B Q4_K_M for anything multilingual. Most quantized GGUF weights come straight from Hugging Face model cards. Total disk used by the model library is about 280GB. Older models are kept around because they sometimes win on specific benchmarks worth caring about.

The whole setup gets driven over the local network from a laptop, and over a private tunnel when remote. A workstation running this load has, in practice, been more reliable than any cloud service of comparable cost. Months without an unplanned restart, with the occasional restart caused by an OS update applied at the wrong moment.

A 64GB M-series workstation is the cheapest serious local LLM machine you can buy in early 2026, and for most independent developers or small teams it's enough. It runs 88B-class models with usable quantization, runs small models at API-tier throughput, and pays for itself in privacy and flat-cost experimentation if not in raw dollars.

If you can afford the capital cost and you have workloads that benefit from local (privacy, latency, bulk processing) the math works. Pair the local rig with a frontier API account for the calls that actually need frontier capability. The two-tier setup is the right architecture for serious solo or small-team work in 2026.

If you can't afford the hardware, or if your workloads don't fit the categories above, stick with the API. The convenience of not running your own infrastructure is real and worth paying for at small scale. The crossover threshold is roughly $80 to $100 per month in API spend. Below that, don't bother. Above it, run the numbers.