Running models on your own machine

Running a large language model locally is no longer only a hobby project. Performance still changes with the chip, memory bandwidth, runtime build, quantization, context length, prompt shape, and background load, which makes comparisons easy to misread. This guide sticks to decisions that transfer between machines: memory sizing, runtime choice, total cost, and which workloads belong locally. For a source-linked starting shortlist and a visible Q4 weight-headroom check, use the Local AI reference.

A high-memory Apple Silicon workstation is one possible local setup; an NVIDIA workstation, CPU server, or ordinary laptop is another. Performance does not scale predictably between those architectures, so this page does not claim a benchr test result for any one machine. If you want to choose an actual model, see the open-weight tier and small language models, then confirm the model artifact size and runtime compatibility in the linked project documentation.

Apple Silicon commonly uses Ollama, llama.cpp, or MLX; NVIDIA deployments may use vLLM, TensorRT-LLM, llama.cpp, or other serving stacks. Treat any third-party throughput number as configuration-specific and reproduce it on your planned system before making a purchase.

The software worth using

The four runtimes below represent different deployment layers. The recommendation is editorial and based on their documented design and public ecosystem, not on an unpublished benchr test.

Ollama is the runtime to recommend to anyone starting today. A small Go binary that runs as a background service, manages model downloads via a registry-style command, and exposes a clean HTTP API on port 11434. The defaults are sensible and the model library is broad and current. Installation is a single command, and a competent user can go from cold start to talking with a local model in under five minutes.

llama.cpp is the runtime to use for production. A C++ project with hand-optimized kernels for Apple Silicon, NVIDIA, AMD, and CPU. It's way faster than Ollama on the same model in some configurations, and it exposes parameters Ollama hides. The cost is that it needs manual compilation, manual model file management, and more documentation reading than is strictly fun. Ollama is built on top of llama.cpp, so this isn't a rejection of Ollama so much as a different layer of the same stack.

LM Studio wraps local inference in a desktop GUI. It is designed for browsing models and comparing them side by side. For GUI-first users, it is a friendlier entry point than compiling or operating a server process.

MLX is Apple's first-party machine learning framework with Apple Silicon support. Its architecture makes it a relevant option for Apple-only deployments, while its ecosystem and integrations differ from llama.cpp-based stacks. Compare current model support and reproduce performance on your own configuration before choosing it for speed.

Memory planning by model size

The ranges below are editorial planning estimates, not benchr measurements. Quantized artifacts vary, and a runtime also needs memory for the KV cache, context, operating system, and other processes. Use the actual model-file size as the starting point and leave headroom.

Editorial memory-planning ranges; verify the published model artifact before deployment

Model class	Typical quantization	Planning range	Practical fit	What to verify
3B–9B	4-bit or 5-bit	Roughly 3–8GB plus overhead	Modern laptops and desktops	Artifact size, context memory, supported backend
14B–32B	4-bit or 5-bit	Roughly 10–24GB plus overhead	Higher-memory laptops or workstations	Memory bandwidth and sustained load
70B class	4-bit or 5-bit	Often above 40GB plus overhead	High-memory workstation or server	Exact artifact, KV cache, usable headroom

Do not compare a small local model with a frontier API on throughput alone. They solve different classes of work, and both local speed and API speed change with configuration and load. Measure time to first token, sustained output rate, task quality, and failure rate on the same representative prompts.

A frontier API and a local model solve different problems. Local inference can keep data on your network and remove a per-token invoice, but hardware, power, maintenance, and review still carry a cost.

Relative local-deployment complexity

Editorial planning guide, not measured performance. Verify the exact artifact, quantization, context setting, and runtime before buying hardware.

The break-even point with an API cannot be universal. It depends on purchase price, useful life, electricity, maintenance, engineering time, workload shape, utilization, and the API calls replaced by the local model.

The cost-benefit, written down without selling anything

Use this calculation: monthly local cost = purchase price ÷ useful months + electricity + storage + maintenance + engineering time. Compare it only with the portion of your API bill a local model can replace at acceptable quality. This is an illustrative formula, not a claim about a private benchr bill. For the API side, see price per use case.

• Control over where inference data is processed. A correctly isolated local stack can keep sensitive material inside an approved network, but local weights alone do not prove that: review telemetry, remote tools, model downloads, logging, backups, monitoring exports, administrator access, and every connected service.
• No internet round trip. Local inference can reduce network-dependent delay, but model compute still takes time; measure end-to-end latency on your own setup.
• Predictable infrastructure cost at sustained volume. More requests do not create a per-token invoice, but they still consume electricity, capacity, storage, and operator time.

If those properties matter to your work, local infrastructure may justify itself even when direct cash cost is close. If they do not, the managed API remains the simpler choice.

One caveat worth stating plainly: interactive demos do not establish behavior under sustained 24/7 load. Fan noise, thermal throttling, concurrency, queueing, and recovery after failure all need a soak test on the hardware you intend to deploy.

Where local pays off

Three workloads where local is the right pick in 2026, named specifically.

Structured extraction from inbound documents: support emails, contract drafts, application forms, and the like. A small local model may be a good candidate when the schema is narrow and the data must stay on your network. Do not assume a fixed accuracy gap: compare it with the API alternative on a labeled sample from your own workload.

Bulk content rewriting against a fixed corpus — hundreds of feature blurbs and product descriptions, microcopy passes, anything where running every iteration through an API would quietly add up. The local model produces the drafts and you edit them, so the only cost is time you're already spending.

Speculative experimentation an API budget might discourage. Local capacity removes a per-token invoice, but it is not costless: account for machine time, electricity, review, and the risks of training on synthetic examples.

Where local isn't the right answer

Anything that needs frontier capability. The local models, even the largest ones that fit on a 64GB machine, stay clearly behind Claude Opus 4.7 and GPT-5 on hard reasoning and multi-file code understanding, and on the kind of voice-sensitive writing where the model's tone matters. Push a local 70B model at work that belongs on the frontier and you spend the day fighting the gap.

Anything heavily multimodal. The local image-understanding story is way weaker than the closed APIs, and for vision tasks Gemini 3.1 Pro Preview through its API is simply where the work should go. See the multimodal ranking for the full picture.

Anything where your team has no appetite to maintain the setup. Local means handling updates and debugging memory pressure, and reading the changelog whenever llama.cpp ships a breaking change. A team that doesn't want to be its own ops team should stay on the API and skip the local setup entirely.

A reference setup

A reference architecture can use Ollama for ad-hoc work and llama.cpp or another documented server runtime behind authentication, queueing, monitoring, and explicit model-version pinning for a service. Keep only the models your evaluation set justifies, and obtain quantized artifacts from publishers whose provenance and license you can verify. The Phi-4 mini model card, Hugging Face model cards, and each runtime's documentation are the relevant starting points.

Remote access should use a private network or authenticated gateway. Production reliability depends on your monitoring, capacity planning, update process, and recovery tests; this article does not claim a private benchr uptime record.

A high-memory workstation can make larger quantized models practical, while small models fit a wider range of laptops and desktops. There is no single “cheapest serious” machine without defining the model, context, concurrency, quality target, and regional price.

If you can afford the capital cost and you have workloads that benefit from local — privacy-bound data, latency-sensitive paths, bulk processing — the math works. Pair the local rig with a frontier API account for the calls that need frontier capability. For serious solo or small-team work in 2026, that two-tier setup is hard to beat.

If the hardware or operational work is not justified by your workload, stick with the API. The convenience of managed infrastructure is genuine and often worth paying for at small or irregular scale. Use the total-cost formula above and your own evaluation set; this page does not assert a universal dollar crossover.