What quantization should I use for a local LLM?

Q4_K_M is the usual default: llama.cpp measures it at 4.89 bits per weight, about a third of the 16-bit size, with small quality loss. Use Q5_K_M (5.70 bits) or Q6_K (6.56 bits) if you have spare memory and care about accuracy, Q8_0 (8.50 bits) when memory is plentiful, and drop to Q3_K_M (4.00 bits) or IQ-quants only to squeeze a larger model into a fixed budget.

Can I run a 70B model on a 24 GB GPU?

Not fully on the GPU. A 70B model at Q4_K_M is about 43 GB of weights, so on a 24 GB card most layers spill to system RAM and generation slows sharply. Options are two 24 GB GPUs, a 48 GB workstation card, an Apple Silicon Mac with 64 GB or more of unified memory, or a smaller model in the 27B to 32B range, which fits at Q4_K_M with room for context.

Does context length use VRAM?

Yes, and it grows linearly with the number of tokens. KV cache bytes per token equal 2 times the number of layers times the number of key-value heads times the head dimension times the bytes per value. For Qwen3-8B (36 layers, 8 KV heads, head dimension 128) at 16-bit that is 147,456 bytes per token, about 4.5 GB at 32,768 tokens. Quantizing the KV cache to 8-bit halves it.

Quick Answer

How Much VRAM Do You Need to Run a Local LLM? (2026 Guide)

Q: How much VRAM do you need to run a local LLM?

Roughly the model's parameter count in billions times the bits per weight divided by 8, in gigabytes, plus room for the context (KV cache) and about 1 to 2 GB of overhead. At the common Q4_K_M quantization (about 4.9 bits per weight in llama.cpp) an 8B model needs about 5 GB for weights, a 32B model about 20 GB and a 70B model about 43 GB. Add the KV cache for your context length: for a Qwen3-8B-class model that is about 144 KB per token at 16-bit, or roughly 4.5 GB at 32,000 tokens.

Published: November 2, 2026

The short answer

Memory needed ≈ weights + KV cache + 1–2 GB overhead.

Weights (GB) ≈ parameters (billions) × bits per weight ÷ 8.
KV cache (bytes) = 2 × layers × KV heads × head dim × bytes per value × tokens.

At the default Q4_K_M quantization, budget about 0.6 GB per billion parameters for weights, then add context. That gives these practical floors (as of November 2026):

Model size	Q4_K_M weights	+ 8K context (typical)	Fits on
3–4B	~2–2.5 GB	~3–4 GB	6 GB GPU, any 8 GB Mac
7–8B	~5 GB	~6–7 GB	8 GB GPU, 16 GB Mac
14B	~9 GB	~10–11 GB	12 GB GPU
27–32B	~17–20 GB	~19–23 GB	24 GB GPU, 32 GB Mac
70B	~43 GB	~45–48 GB	2× 24 GB GPUs, 48 GB card, 64 GB Mac
gpt-oss-120b (MoE)	—	—	One 80 GB GPU (OpenAI)

Context figures assume a grouped-query-attention model and a 16-bit KV cache; long contexts need far more (see step 3).

Step 1: Pick the quantization

llama.cpp’s quantize README publishes measured bits per weight for each GGUF format:

Format	Bits / weight	8B model size*	When to use
Q3_K_M	3.996	3.74 GiB	Squeezing a bigger model into fixed memory
IQ4_XS	4.460	4.17 GiB	Slightly smaller than Q4_K_M, similar quality
Q4_K_M	4.894	4.58 GiB	Default
Q5_K_M	5.704	5.33 GiB	Spare memory, better accuracy
Q6_K	6.563	6.14 GiB	Near-lossless for most tasks
Q8_0	8.501	7.95 GiB	Memory is not a constraint
F16	16.0	14.96 GiB	Reference / fine-tuning

*Sizes from the README’s 8B benchmark model. The same README lists Llama 3.1 sizes at Q4_K_M: 8B 4.9 GB, 70B 43.1 GB, 405B 249.1 GB (from 32.1 GB, 280.9 GB and 1,625.1 GB originals).

Rule: if a model only fits below Q3, pick a smaller model at Q4_K_M instead — it usually answers better.

Step 2: Compute the weights

Multiply. A 32B model at Q4_K_M: 32 × 4.894 ÷ 8 ≈ 19.6 GB. At Q6_K: 32 × 6.563 ÷ 8 ≈ 26.3 GB — no longer fits a 24 GB card.

Mixture-of-experts models need memory for all parameters, not just the active ones: active parameters set speed, total parameters set memory. OpenAI states gpt-oss-120b runs on a single 80 GB GPU and gpt-oss-20b within 16 GB. Frontier-size open MoE models are another class: Mistral Large 4’s NVFP4 checkpoint is about 610 GB, which is multi-GPU server territory.

Step 3: Add the KV cache for your context

Read four values from the model’s config.json: num_hidden_layers, num_key_value_heads, head_dim, and the cache precision you will run (2 bytes for 16-bit, 1 for 8-bit).

Worked example, Qwen3-8B (36 layers, 8 KV heads, head dimension 128):

Per token: 2 × 36 × 8 × 128 × 2 bytes = 147,456 bytes (~144 KiB)
8,192 tokens: ~1.1 GiB
32,768 tokens: ~4.5 GiB
With an 8-bit KV cache: half of each.

So an 8B model at Q4_K_M with a 32K context needs about 4.6 + 4.5 + 1 ≈ 10 GB, not 5. Long context, not the weights, is what breaks most “it should fit” estimates. Running several requests in parallel multiplies the cache by the number of concurrent sequences.

Step 4: Check the hardware type

NVIDIA/AMD GPUs: VRAM is a hard wall. Layers that do not fit offload to system RAM and run several times slower.
Apple Silicon: unified memory is shared, and macOS keeps part of it for the system, so do not plan to use all of it for the model. A 64 GB Mac comfortably runs 70B at Q4_K_M with modest context.
CPU only: works for small models; memory bandwidth, not size, limits speed.

Step 5: Leave headroom

Add 1–2 GB for the runtime, CUDA/Metal buffers and the desktop. If the total is within 10% of your memory, drop one quantization level or halve the context.

Self-host or call an API?

If the model you want needs a GPU you do not own, compare with hosted prices first: gpt-oss-120b, for example, is $0.15 input / $0.60 output per million tokens on Groq, Together and Fireworks (November 2, 2026). Our self-host vs API break-even guide does that math, and best local LLM to run picks models per hardware tier. Token rates for hosted models are in our current API prices table.

Last verified: November 2, 2026.