What is the best open-weight model for a single GPU in 2026?

For a 24 GB card, Qwen3.6-27B is the strongest all-around default and the best agentic coding model that fits. For 16 GB, gpt-oss-20b at roughly 12 GB in Q4_K_M is the best balance of capability and headroom. For long-context agent work on 24 GB, Xing4.0-29B-A4B — released September 22, 2026 under Apache 2.0 with 256K native context — is the specialist pick. Below 12 GB you are choosing among 7-9B class models, where quality drops sharply for agentic tasks.

How much VRAM do I need to run a local LLM in 2026?

The working rule at 4-bit quantization is about 0.6 GB of VRAM per billion total parameters, plus KV cache for your context. A 20B model needs roughly 12 GB, a 27-29B model roughly 16-18 GB, before any context. Add 2-8 GB for a usable KV cache depending on how long your prompts are. That makes 24 GB the practical sweet spot in 2026: it runs the 27-30B tier with real context headroom. 16 GB runs the 20B tier well. 12 GB limits you to roughly 14B and below.

Why can't I run a 300B mixture-of-experts model on my 24GB GPU?

Because mixture-of-experts saves compute, not memory. A model like MiMo-V2.6-Flash has around 309 billion total parameters with about 15 billion active per token. The active count determines generation speed; the total count determines how much memory must hold weights. All 309 billion parameters have to be resident somewhere, which is datacenter territory even at aggressive quantization. Active parameters tell you how fast, total parameters tell you whether it fits.

Is quantization worth it, or does it ruin quality?

4-bit quantization is worth it for almost every local deployment in 2026. Modern methods — Q4_K_M in GGUF, AWQ and GPTQ for GPU inference, FP8 where the vendor publishes it — lose a few percent on benchmarks while cutting memory by roughly 4x versus FP16. The practical consequence is that a 4-bit 27B model beats an 8-bit 13B model on the same card almost every time. Quantize down to fit a bigger model, not the other way around. Below 4-bit, degradation becomes visible on reasoning and code.

What hardware should I buy for local LLMs in 2026?

A used RTX 3090 at 24 GB remains the best value per gigabyte of VRAM and runs the entire 27-30B tier. An RTX 4090 adds speed at a large price premium; an RTX 5090 at 32 GB buys context headroom rather than a new model tier. Apple Silicon with 32 GB or more of unified memory is competitive for GGUF inference and unbeatable on power draw and noise, at lower tokens per second than a discrete GPU. VRAM capacity determines which models you can run at all; memory bandwidth determines how fast they run. Buy capacity first.

Quick Answer

Best Open-Weight Models for a Single GPU 2026 (Ranked)

Published: September 26, 2026

The short answer

On 24 GB, run Qwen3.6-27B. On 16 GB, run gpt-oss-20b. For long-context agent work on 24 GB, run Xing4.0-29B-A4B. The binding constraint in 2026 is total parameters at your quantization, not architecture or benchmark scores — and the most common mistake is assuming a mixture-of-experts model with few active parameters is small. It is not. Active parameters set speed; total parameters set whether it loads.

The ranking

Rank	Model	Total / active	4-bit VRAM	Context	License	Best for
1	Qwen3.6-27B	27B dense	~16-18 GB	Long	Qwen permissive	Strongest all-around on 24 GB
2	Xing4.0-29B-A4B	29B / 4B MoE	~15-18 GB	256K → 512K	Apache 2.0	Long-context agents, tool use
3	gpt-oss-20b	21B / ~3.6B MoE	~12 GB	Standard	Apache 2.0	16 GB cards, mixed fleets
4	Gemma 4 (mid tier)	Dense	~14-18 GB	Long	Gemma terms	Safety-tuned general use
5	Mistral mid-tier open weights	Dense	~14-16 GB	Long	Apache 2.0	EU data residency, fast
6	DeepSeek distills (14B class)	14B dense	~9 GB	Standard	MIT-family	12 GB cards, reasoning
—	MiMo-V2.6-Flash	~309B / ~15B	Datacenter	1M	MIT	Does not fit a consumer GPU
—	MiMo-V2.6-Pro	~1.02T / 42B	Datacenter	1M	MIT	Does not fit a consumer GPU

The last two rows are there because they are the most-searched and most-misrecommended models in the tier. They are excellent and they are not desktop models.

Sizing: the only formula you need

At 4-bit quantization: VRAM ≈ 0.6 GB × total parameters in billions, + KV cache.

Card	Comfortable model size	Realistic picks
12 GB (3060 12GB, 4070)	up to ~14B	DeepSeek 14B distill, 9B class
16 GB (4060 Ti 16GB, 5060 Ti 16GB, A4000)	up to ~21B	gpt-oss-20b
24 GB (3090, 4090)	up to ~30B	Qwen3.6-27B, Xing4.0-29B-A4B
32 GB (5090)	up to ~35B, or 30B with long context	Same tier, more KV cache
32-64 GB Apple unified	up to ~35B via GGUF	Same tier, lower throughput

KV cache is the part people forget. Weights are a fixed cost; context is a running one that scales with prompt length and grows fast at long contexts. On a 24 GB card with ~17 GB of weights, your ~7 GB of remaining memory is the real limit on how long a prompt you can serve — tens of thousands of tokens, not the model’s headline 256K.

Practical method: load at 4-bit, set context to 32K, run your longest real workload, raise context until you hit an out-of-memory error, then back off 25%.

Why 24 GB is the 2026 sweet spot

The capability cliff in open weights sits between roughly 20B and 27B total parameters. Below it, models are useful for summarization, classification and simple code completion, and unreliable for multi-step agentic work — they lose the thread across tool calls. Above it, agentic reliability arrives.

24 GB is the cheapest way to sit above that cliff with room for context. A used RTX 3090 is still the best VRAM-per-dollar option on the market and runs the entire 27-30B tier. The 4090 buys throughput at a large premium; the 5090’s 32 GB buys context, not a new tier.

Apple Silicon deserves its own note. A 36 GB or 48 GB Mac runs the same GGUF quants at lower tokens per second, in silence, at a fraction of the power draw, with no driver work. For an always-on local agent rather than an interactive coding session, that trade is often correct.

The mixture-of-experts trap

This is the single most expensive misunderstanding in local LLM setup in 2026, so it is worth stating twice.

MoE models save compute, not memory. MiMo-V2.6-Flash activates about 15B of ~309B parameters per token. That makes it generate at roughly 15B-dense speed — genuinely fast. It does not make it a 15B model. All ~309B parameters must be resident. Even at 4-bit that is over 180 GB.

The same applies upward: MiMo-V2.6-Pro is ~1.02T total with 42B active, and Xiaomi’s own guidance is that it needs a multi-GPU server. It is an API model.

Where MoE does help you locally is at the small end. Xing4.0-29B-A4B at 29B/4B and gpt-oss-20b at 21B/~3.6B are MoE models sized for consumer memory, and they are faster than dense models of the same footprint. That is the combination to look for: MoE architecture with a total parameter count under 30B.

Quantization: go lower, go bigger

The counterintuitive rule that holds consistently: a 4-bit large model beats an 8-bit small model on the same card.

Format	Memory vs FP16	Quality cost	When to use
FP16 / BF16	1x	None	Server, benchmarking
FP8	~0.5x	Negligible	vLLM/SGLang on 24 GB+
8-bit (Q8)	~0.5x	Negligible	When you have spare VRAM
4-bit (Q4_K_M, AWQ, GPTQ)	~0.25x	Low single-digit %	Default for consumer GPUs
3-bit and below	~0.2x	Visible on reasoning and code	Only to squeeze a tier up

Prefer official quants where the vendor ships them. Xing4.0-29B-A4B published FP8 and GGUF builds alongside the base weights on release day, which removes the guesswork about which community quant is faithful.

Runtime picks

Runtime	Best for	Notes
Ollama	Getting running in five minutes	GGUF, one-command model pulls
LM Studio	Desktop GUI, model discovery	Good quant browser, Apple Silicon friendly
llama.cpp	Maximum control, CPU offload	Everything else is built on it
vLLM	Serving, throughput, FP8	Needs more VRAM headroom; server-oriented
SGLang	Structured output, agent workloads	Strong for tool-calling loops

For agent workloads specifically, vLLM or SGLang with FP8 on a 24 GB+ card will hold up under concurrent tool calls better than Ollama, which is tuned for single-user interactive use.

Decision rule

You have 24 GB and want one model: Qwen3.6-27B at Q4_K_M.
You have 24 GB and are building a long-context agent: Xing4.0-29B-A4B, FP8 on vLLM or Q4 GGUF on Ollama.
You have 16 GB: gpt-oss-20b, Q4_K_M.
You have 12 GB: a 14B-class distill; accept that agentic reliability is limited.
You are shopping: used RTX 3090 for best value; Apple Silicon 36 GB+ for a quiet always-on box.
You saw a 300B MoE recommended for a desktop: ignore it. Check total parameters, not active.

Head-to-head detail on the top three is in Xing4.0-29B-A4B vs Qwen3.6-27B vs gpt-oss-20b. If your ceiling is 16 GB and you want a much larger model anyway, the ternary-quantization route is covered in how to run a 27B LLM on a 16GB laptop.

Last verified: September 26, 2026. VRAM figures are 4-bit estimates before KV cache; actual usage varies by quantization method and runtime.