Running language models locally

Local inference comes down to two separate questions: does the model fit, and how fast does it run? The first is a memory capacity question, the second a memory bandwidth question. Three open source tools cover almost every setup, and the hardware math is really those two formulas. This note gives both and maps them onto the tools.

Capacity and speed are separate questions

Most local LLM questions trace back to one confusion: memory capacity is mixed up with speed. The two are independent.

Capacity, in gigabytes, decides what can be loaded. Speed, in tokens per second, is set by memory bandwidth, the rate at which the hardware can stream bytes out of memory. Adding more RAM never makes generation faster. It only lets a larger model load, which then runs at whatever rate the bandwidth allows.

This is why a 64 GB DDR4 server and an 8 GB RTX 4060 laptop answer different questions: the server fits a 70B model at a few tokens per second, the laptop runs a 7B model quickly and cannot load anything larger.

Weight size and quantization

The first term in both capacity and speed is the size of the weights. A model with $N$ parameters in a precision with $b$ bytes per parameter occupies:

Precision sets $b$: FP16 uses 2 bytes, INT8 roughly 1, and 4-bit quantization roughly 0.5. Because decoding reads every weight once per token, halving $b$ halves the memory and roughly doubles the speed ceiling. This is the whole point of quantization.

GGUF is the single-file format that packages quantized weights, the tokenizer, and metadata. Q4_K_M is the default for most hardware: the quality loss versus FP16 is small on ordinary tasks and becomes visible mainly on heavy reasoning. Q5_K_M and Q8_0 are the upgrades when memory allows, and IQ4_XS is the step down when it does not.

The speed formula

Generation happens in two phases with two different bottlenecks. Prefill processes the whole prompt at once as large parallel matrix multiplications, so it is compute-bound and fast; it sets the time to the first token. Decode produces one token at a time, and each token requires reading every active weight out of memory. Decode is bandwidth-bound and sets the perceived speed.

For decode, the ceiling is:

The efficiency factor accounts for KV cache reads, kernel overhead, and the fact that no hardware sustains its rated bandwidth; realistic values run from 0.5 to 0.8, with 0.7 a working default. The weight bytes per token is the active weight size: the full model for a dense model, only the routed experts for a mixture of experts, which is why the transformer note treats active parameters as the right unit for running cost.

Reference bandwidths: dual-channel DDR4 moves about 25 to 50 GB/s, dual-channel DDR5 about 60 to 90 GB/s, Apple unified memory from roughly 70 GB/s on the base chips to 400 to 800 GB/s on the Max and Ultra tiers, and consumer GPUs from roughly 270 GB/s (RTX 4060) to 1000 GB/s (RTX 4090).

An 8B model at Q4_K_M occupies about 4.9 GB. On DDR5 at 90 GB/s the ceiling is $90 / 4.9 \approx 18$ tokens/s and the realistic rate lands near 10 to 15. On an RTX 4090 at 1008 GB/s the ceiling is $1008 / 4.9 \approx 200$ tokens/s and the realistic rate lands near 90 to 140. On an Apple M4 Max at roughly 500 GB/s the ceiling is about 100 tokens/s and the realistic rate near 40 to 80.

The formula also runs in reverse to size hardware for a speed target:

To reach 30 tokens/s with that 4.9 GB model, the machine needs roughly $30 \times 4.9 / 0.7 \approx 210$ GB/s, which no DDR5 system reaches. The target requires a GPU or an Apple chip. This is the arithmetic behind every slow-local-model post: the CPU is not the weak part, the RAM bus is.

Prefill and time to first token

The decode formula sets the rate after generation starts. Before any token appears, the prompt must be processed, and that phase has its own ceiling. Prefill runs every prompt token through the model in parallel as one large matrix multiply, so it is compute-bound rather than bandwidth-bound:

The factor of 2 counts the multiply and add per parameter per token, and realistic compute efficiency runs from 0.3 to 0.5. Time to first token (TTFT) follows directly:

The two regimes favor different hardware, and the gap between them can be large. An 8B model prefills at roughly 7,200 tokens/s and decodes at roughly 150 tokens/s on an RTX 5090, a roughly 45-fold spread, while an Apple M5 Max prefills near 400 tokens/s and decodes near 100 tokens/s, a 4-fold spread. The workload decides which number matters: chat with short prompts and long answers is felt through decode, while agentic and coding workloads, which re-send a long context on every turn, are felt entirely through TTFT. This is why a long-context prompt can sit for seconds before the first token even on hardware that generates quickly afterward.

What fits where

Total memory is three terms:

The KV cache is a function of context length, and the KV caching note derives it: roughly 128 KB per token for a typical 32-layer grouped-query model, so a 4,096-token context needs about 512 MB and a 32K context about 4 GB. The overhead of activation buffers and the runtime is 1 to 3 GB, larger for Python stacks than for llama.cpp.

The practical tiers, at Q4_K_M:

Context length is the forgotten variable. A model that fits at 8K context with room to spare can hit the memory wall at 32K, and capping context is often a cheaper fix than buying a bigger card.

The capacity wall can be pushed past when the model does not need to be fast. Streaming weights from an NVMe disk instead of holding them in memory lets a model larger than the combined RAM and VRAM run, because pages are read from disk on demand. llama.cpp does this by default: --mmap is on, --mlock is off, and the --fit flag sizes buffers to the hardware. Generation drops toward disk bandwidth, a few tokens per second, but the ability to test a model without owning hardware for it is useful enough that the community runs evaluations and light workloads this way.

CPU-only inference

CPU-only inference is fully supported, and it is where llama.cpp is strongest, with AVX2, AVX-512, and ARM NEON kernels. The constraint is the same bandwidth wall as everywhere else, so the lesson is that the RAM bus, not the core count, sets the ceiling. Dual-channel DDR5 is roughly twice as fast as DDR4 for this workload. Beyond about eight threads, adding cores buys little, because the memory bus saturates first.

Expectations on a modern desktop: a 7B model at Q4 runs at roughly 5 to 15 tokens/s, comfortable at reading speed but not for bursty chat. A 70B model fits in 64 GB of RAM and runs at roughly 2 to 4 tokens/s, usable for offline summarization and not for conversation. A small model such as a 3B runs near reading speed even on modest hardware.

Three flags matter on CPU. --mlock and --no-mmap control whether the model stays resident in RAM and how it loads, and -t sets the thread count, where physical cores rather than logical threads is the better value.

llama.cpp flags that matter

The two with the biggest effect are the ones most often left wrong. --ctx-size pre-allocates the entire KV cache at startup, so a generous value quietly reserves gigabytes before any token is generated, and a context set larger than the hardware can hold causes swap and an apparent collapse in speed. --cache-type-k q8_0 --cache-type-v q8_0 halves the cache, which is often the difference between fitting and not, but the quality is not uniformly free: q4_0 measurably degrades long-context agentic and coding output, with malformed tool calls the usual symptom, and some architectures, such as Gemma, are sensitive even at q8_0. Test the configuration before trusting it. Both flags follow from the memory accounting in the KV caching note .

Speculative decoding is one flag but two models: a small draft model proposes several tokens and the large model verifies them in a single parallel pass, which typically lifts generation by 1.5 to 2 times on bandwidth-bound hardware. Some builds default the --spec-type setting to none, so a draft model loads into memory while generation stays autoregressive, and the draft and its type must both be set for the speedup to engage.

Putting the levers together, a starting point for an 8 GB GPU running a 4B Q4 model as a local OpenAI-compatible server:

1llama-server -hf unsloth/Qwen3.5-4B-GGUF:Q4_K_M --no-mmproj -ngl 99 --ctx-size 8192 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn -t 8 -b 512

Every value is a decision from the tables above, not a universal optimum: the KV budget, offload count, and thread count change with the hardware and the context needed.

The tool ecosystem

Every mainstream local tool runs the same GGUF files through the same engine, llama.cpp, so the choice is about the layer on top, not the math underneath.

• llama.cpp is the raw engine: a native binary with no runtime, roughly 20 MB for the Windows CPU build, the fastest option, the newest model support, and the only one that targets CPU inference as a first-class workload. It contains no model; the gigabytes are in the GGUF file. The cost is that everything is manual, from downloading the right build to passing flags.
• llamafile inverts that split: it bundles the engine and the weights into one cross-platform executable using Cosmopolitan libc, so a single file runs on Linux, macOS, and Windows with no installation, the way to hand a model to someone who should not have to configure anything. The bare runner is about 60 MB and takes an external GGUF; the bundled form grows to the size of the model. Windows cannot execute files over 4 GB, so large bundled models revert to the two-file form there. The same packaging produces whisperfile, a single-file speech-to-text tool.
• Ollama wraps llama.cpp in a daemon with a model registry and an OpenAI-compatible API on port 11434. ollama pull handles downloads and quantization selection. The convenience costs roughly 10 percent in throughput and a gigabyte or so of resident memory.
• LM Studio is a desktop application with a Hugging Face browser, a chat interface, and a one-click server. On Apple silicon its MLX backend is the fastest option for many models. It is the easiest on-ramp and the least suited to automation.
• vLLM is the production answer for multi-user GPU serving: PagedAttention and continuous batching deliver far higher aggregate throughput than the single-user tools. It is Python and CUDA centered, ignores GGUF, and is overkill for a single machine. SGLang is the main alternative for latency-sensitive or day-zero-model serving, and the older TGI was archived in early 2026, so guides still pointing at it are stale.
• llama-swap is a proxy that starts and stops backend processes on demand, routing model names to different engines from one config, useful when an application needs several specialized models.
• Open WebUI and Jan are chat frontends that sit on top of any of the servers.

The practical progression is LM Studio or Ollama to start, and llama.cpp or vLLM once the abstraction gets in the way.

Measuring

The formulas give the ceiling, not the result. llama-bench -m model.gguf -n 128 measures prompt processing and generation on the actual machine, and --verbose prints the KV cache size at startup, the number to check when the context math stops matching reality. Measured generation typically lands between 50 and 80 percent of the bandwidth ceiling, and the gap is the efficiency factor plus whatever was left misconfigured. Run the measurement, compare it to the ceiling, and the difference tells which problem remains.

The measured numbers feed the one economic formula worth knowing. Cost per million output tokens on owned hardware:

A card generating 100 tokens/s for a dollar an hour works out to about $2.8 per million tokens, comfortably below the single-digit API rates for frontier models. Two thresholds calibrate the feel of the numbers: interactive chat wants a time to first token under two or three seconds, and generation below about 10 tokens/s drops below reading speed.