How do I calculate how much VRAM an LLM needs?

Multiply parameters by bytes per parameter for your precision: 2 bytes at FP16/BF16, 1 byte at INT8, 0.5 bytes at INT4. A 7B model is therefore about 14 GB at FP16, 7 GB at INT8 and 3.5 GB at INT4. Then add KV cache and activation overhead — 15 to 25 percent at short context, and potentially more than the weights themselves at 100K+ tokens.

Does quantisation hurt model quality?

INT8 is close to lossless for most tasks. INT4 is where measurable degradation begins, and it hits structured output, long-chain reasoning and tool-call formatting hardest — exactly the behaviours agent loops depend on. Benchmark your own task at both precisions before assuming 4-bit is free; a 4-bit 12B is not automatically better than an 8-bit 7B.

Can I run a mixture-of-experts model on a single GPU?

Only if the total parameter count fits, not the active count. K2-Horizon-MoVA-36B-A4B activates 4B parameters per token but still requires all 36B resident in memory. MoE reduces per-token compute, not memory footprint, so on a single consumer card a dense model of equivalent memory cost is usually the better choice.

Why does my model run out of memory at long context when it fit at short context?

KV cache grows linearly with sequence length and with the number of concurrent requests. Weights are a fixed cost; KV cache is not. A 7B that fits comfortably at 4K tokens can exhaust the same card at 128K, and batching several long-context requests multiplies the problem. Size for your longest realistic context and your real concurrency, not for a single short prompt.

Quick Answer

How to Choose an LLM Size for Your GPU (2026 Guide)

Published: September 7, 2026

The Short Answer

Three numbers decide whether a model fits your GPU: weights, KV cache, and overhead. Most people compute the first and get surprised by the second.

VRAM needed ≈ (params × bytes_per_param)     ← weights
            + KV cache (grows with context × concurrency)
            + 15–25% activation & runtime overhead

Everything below is how to fill in those terms without guessing.

Step 1: Compute the Weight Cost

Bytes per parameter, by precision:

Precision	Bytes/param	Quality
FP16 / BF16	2	Reference
INT8	1	Near-lossless for most tasks
INT4	0.5	Measurable degradation begins

Multiply through:

Model size	FP16	INT8	INT4
3.7B	~7.4 GB	~3.7 GB	~1.9 GB
7B	~14 GB	~7 GB	~3.5 GB
9B	~18 GB	~9 GB	~4.5 GB
12B	~24 GB	~12 GB	~6 GB
32B	~64 GB	~32 GB	~16 GB
36B (MoE total)	~72 GB	~36 GB	~18 GB

Done when: you have a weight figure for at least two precisions of your candidate model.

Step 2: Add the KV Cache — The Term That Ruins Plans

The KV cache stores attention keys and values for every token in the context. It grows linearly with sequence length and linearly with concurrent requests.

This is why a model that ran fine in testing dies in production. Your test was one 2,000-token prompt. Production is eight concurrent 60,000-token agent conversations.

Practical guidance rather than a formula (exact size depends on layer count, head dimension, grouped-query attention and cache precision):

Context	KV cache relative to a 7B model’s weights
4K tokens	Small — a few percent
32K tokens	Noticeable — often 10–25%
128K tokens	Comparable to the weights
512K tokens	Can exceed the weights substantially

Modern small models advertise big windows — K2-Horizon-7B carries 512K, and DeepSeek V4 Flash defaults to 1M. Those windows are real, and they are not free. A 512K-context 7B is a different memory problem from a 4K-context 7B.

Two mitigations that work:

Quantise the KV cache (8-bit KV is widely supported in vLLM and SGLang) — often a bigger win than quantising weights further.
Cap max_model_len to your actual longest realistic prompt rather than the model’s maximum. Serving frameworks pre-allocate against this.

Done when: you have sized for your longest realistic context at your real concurrency, not a single short prompt.

Step 3: Decide Your Quantisation Level Honestly

The tempting move is always “quantise harder, run a bigger model.” Sometimes right, often wrong.

INT8 is close to free. For most workloads the quality delta is inside noise. Take it.

INT4 is where things break, and they break in a specific pattern. Degradation concentrates in:

Structured output — JSON that no longer validates
Tool-call formatting — arguments that drift from the schema
Long-chain reasoning — errors compound across steps

Those are precisely the behaviours an agent loop depends on. A chatbot barely notices INT4. An agent notices immediately.

The comparison to run: a 4-bit 12B against an 8-bit 7B at similar memory cost, on your task. Do not assume the larger model wins — at aggressive quantisation it frequently does not, especially for tool use.

Done when: you have measured both configurations on your own evaluation, not on a leaderboard.

Step 4: Handle MoE Correctly

The single most common sizing error in 2026.

A sparse mixture-of-experts model has two parameter counts, and people optimise for the wrong one:

Model	Total	Active/token	You must hold
K2-Horizon-MoVA-36B-A4B	36B	4B	36B
K2-Horizon-375B-A23B	375B	23B	375B
DeepSeek V4 Flash	284B	13B	284B
DeepSeek V4 Pro	1.6T	49B	1.6T

Active parameters determine compute and latency. Total parameters determine memory. MoE gives you the speed of a small model at the memory cost of a large one.

That trade is excellent on a server with abundant VRAM. On a single consumer card it is backwards: you pay 36 GB of memory for 4B worth of throughput advantage, when a dense 12B–32B in the same memory would simply be more capable per byte.

Rule: on one GPU, prefer dense. Consider MoE only when the total fits comfortably.

Done when: you have sized against total parameters for any MoE candidate.

Step 5: Match to Your Card

Assuming short-to-medium context and single-stream inference:

VRAM	Comfortable	Stretch (quantised / short context)
8 GB	3.7B FP16, 7B INT4	9B INT4
12 GB	7B INT8	12B INT4
16 GB	7B FP16, 12B INT8	32B INT4
24 GB	12B FP16, 32B INT4	7B at 128K context
48 GB	32B INT8	36B MoE INT8, 7B at very long context
80 GB+	32B FP16	375B-class MoE needs multi-GPU regardless

Then subtract for context. If you need 128K+, drop one row. If you need concurrency, drop another.

Done when: your chosen configuration fits with at least 15% headroom — running a GPU at 99% allocation is how you discover OOM at the worst moment.

Step 6: Verify Before You Commit

# vLLM: cap context to what you actually need and quantise the KV cache
vllm serve <model> --max-model-len 32768 --kv-cache-dtype fp8

# Ollama: GGUF quantisation is selected by tag
ollama run <model>:7b-instruct-q4_K_M

Then load it, run your longest realistic prompt at your target concurrency, and watch actual VRAM usage. The only sizing number that matters is the one your card reports under your load.

The Short Version

Weights = params × bytes/param (2 / 1 / 0.5).
Add KV cache — it scales with context and concurrency, and can exceed the weights past 128K.
INT8 is nearly free; INT4 costs you tool calls and structured output.
MoE memory = total, not active. Dense wins on one card.
Leave 15% headroom and verify under real load.

Last verified: September 7, 2026.