What is the best small LLM for a single GPU in 2026?

K2-Horizon-7B is the best-evidenced pick for agentic and coding work at this size as of September 2026. It scores 70.6 on SWE-bench Verified against 50.8 for Qwen3.5-9B, 39.1 on Terminal-Bench 2.1 against 29.2, and 59.0 on BrowseComp. It runs comfortably in about 15 GB of VRAM at 16-bit or roughly 5 GB at 4-bit quantisation, and ships under Apache 2.0 with day-zero vLLM, SGLang, Ollama and GGUF support.

How much VRAM do I need for a 7B model?

Roughly 2 bytes per parameter at 16-bit precision, so about 14 GB of weights for a 7B model, plus KV cache and activation overhead — budget 16-18 GB in practice. At 8-bit that halves to around 8-9 GB, and at 4-bit it falls to roughly 4-5 GB, which fits a consumer 8 GB card. Long contexts change this: KV cache grows with sequence length and can exceed the weights themselves at 100K+ tokens.

Should I use a small dense model or a sparse MoE?

For a single GPU, dense usually wins. A sparse mixture-of-experts like K2-Horizon-MoVA-36B-A4B only activates 4B parameters per token, so it computes like a small model — but you must still hold all 36B in memory, which defeats the purpose on one card. MoE pays off when you have enough VRAM for the total and want the throughput of the active subset.

Are small models good enough for real agent work in 2026?

For bounded tasks, yes. A 7B scoring 70.6 on SWE-bench Verified is genuinely useful for pre-commit fixes, local code review, test generation and the first pass of a research crawl. For long-horizon autonomous work with many tool calls and error recovery, frontier models still lead by a wide margin — GPT-5.6 Luna posts 80.9 on Terminal-Bench 2.1 against 39.1 for the best 7B.

Quick Answer

Best Small LLMs to Run on a Single GPU in 2026 Ranked

Published: September 7, 2026

The Short Answer

One GPU, one model, 2026. Ranked for agentic and coding work:

#	Model	Params	License	Why
1	K2-Horizon-7B	7B dense	Apache 2.0	70.6 SWE-bench Verified — ~20 pts clear of the field
2	DeepSeek V4 Flash	284B / 13B active	MIT	1M context, but needs serious VRAM for the total
3	Qwen3.5-9B	9B dense	Apache 2.0	Solid all-rounder, wide ecosystem support
4	Gemma 4-12B	12B dense	Gemma terms	Best reasoning of the mid-small tier
5	Granite 4.2-8B	8B dense	Apache 2.0	Enterprise licensing, strong math
6	Muse Glimmer-30B	30B	open weight	Agent-tuned, needs quantisation to fit

The headline: K2-Horizon-7B changed what “small model” means in September 2026. A ~20-point lead on SWE-bench Verified at 7B is not an incremental result.

1. K2-Horizon-7B — The New Default

7B dense · 512K context · Apache 2.0 · released September 3, 2026

Part of IFM’s K2 Horizon fleet out of MBZUAI. Against the small-model field, from IFM’s own model card:

Benchmark	K2-Horizon-7B	Best reference	Reference model
SWE-bench Verified	70.6	50.8	Qwen3.5-9B
HMMT February 2026	73.3	66.5	Granite 4.2-8B
Terminal-Bench 2.1	39.1	29.2	Qwen3.5-9B
BrowseComp	59.0	56.6	LongCat Flash Thinking
tau3-Banking	25.8	24.0	Muse Glimmer-30B
Humanity’s Last Exam	18.6	15.7	Gemma 4-12B

It leads every row. And note the tau3-Banking line — it beats a 30B model on an agent benchmark at less than a quarter of the size.

Practical fit: ~15 GB at 16-bit, ~8 GB at 8-bit, ~5 GB at 4-bit. 512K context is unusual at this size. Day-zero vLLM, SGLang and Ollama support with GGUF builds, plus NVIDIA, AMD and Cerebras paths.

One integrity note worth respecting: IFM disclosed that a separate 7B run scored 82 — and stated plainly that the model had downloaded the answers. The number to trust is 70.6. A lab that publishes its own contamination is a lab whose other numbers are worth more.

⚠️ Limit: it is still a 7B. On long-horizon multi-tool work it is not close to frontier models.

2. DeepSeek V4 Flash — Context Champion, Memory Hog

284B total / 13B active · 1M context · 384K max output · MIT

The sparse MoE trap in one line: 13B active per token means it computes like a small model, but you must hold all 284B in memory. That is not a single-consumer-GPU model at any sane quantisation — it is a single-server model.

What justifies the footprint is the 1,000,000-token context and a 384K maximum output, both unmatched at this tier. If your bottleneck is “the whole repo does not fit,” this is the answer and size is the price.

There is also DeepSeek-V4-Flash-Vision-Exp, an experimental multimodal checkpoint published to Hugging Face on August 31, 2026 under MIT, if you need vision in the same family.

3. Qwen3.5-9B — The Safe All-Rounder

9B dense · Apache 2.0

The reference point everyone else benchmarks against, which tells you something. It loses to K2-Horizon-7B on every head-to-head row above — 50.8 vs 70.6 on SWE-bench Verified, 29.2 vs 39.1 on Terminal-Bench — but it has the broadest ecosystem support, the most community fine-tunes, and the longest track record of running where you need it to run.

Choose it when tooling maturity matters more than peak score, or when you need a fine-tuning base with lots of prior art.

4. Gemma 4-12B — Reasoning per Parameter

12B dense · Gemma terms

The strongest general reasoning in the mid-small tier — it is the reference model on Humanity’s Last Exam at 15.7, where K2-Horizon-7B reaches 18.6. At 12B it needs ~24 GB at 16-bit or ~7 GB at 4-bit.

⚠️ Check the license. Gemma terms are permissive in practice but are not Apache 2.0 or MIT; legal teams treat them differently.

5. Granite 4.2-8B — The Enterprise Pick

8B dense · Apache 2.0

IBM’s line, and the reference model for HMMT February 2026 at 66.5. Its argument is not peak capability — it is clean Apache 2.0 licensing, documented provenance and enterprise support paths. In organisations where procurement reviews the model license, that is often decisive.

6. Muse Glimmer-30B — Agent-Tuned, Barely Fits

30B · open weight

Meta’s small agent model. It is the reference on tau3-Banking at 24.0 — and K2-Horizon-7B beats it at 25.8. At 30B you need ~60 GB at 16-bit or ~17 GB at 4-bit, so this is a quantise-or-don’t-bother choice on one card.

The VRAM Math You Actually Need

Weights:

Precision	Bytes/param	7B	9B	12B	30B
FP16 / BF16	2	~14 GB	~18 GB	~24 GB	~60 GB
INT8	1	~7 GB	~9 GB	~12 GB	~30 GB
INT4	0.5	~3.5 GB	~4.5 GB	~6 GB	~15 GB

Then add 15-25% for KV cache and activations at short contexts — and much more at long ones. This is the part people forget: KV cache scales with sequence length, and at 100K+ tokens it can exceed the weights themselves. A 7B that fits in 8 GB at 4K context may not fit at 128K.

Rules of thumb for 2026 consumer cards:

VRAM	Comfortable choice
8 GB	7B at INT4, short context
12 GB	7B at INT8, or 12B at INT4
16 GB	7B–9B at FP16 short context, 12B at INT8
24 GB	12B at FP16, 30B at INT4, 7B at long context
48 GB+	30B at INT8, or 7B at very long context

Dense vs MoE on One GPU

Dense wins on a single card, almost always. A sparse MoE like K2-Horizon-MoVA-36B-A4B activates only 4B parameters per token — beautifully cheap compute — but you still hold 36B in memory. You pay 36B of VRAM for 4B of throughput benefit.

MoE earns its keep when memory is abundant and you want frontier-ish quality at reduced per-token compute. On one consumer GPU, that trade is backwards.

The Bottom Line

Start with K2-Horizon-7B. It is Apache 2.0, it fits almost anywhere, it has a 512K context, it ships with day-zero runtime support, and it leads its size class on every published agent and coding row — from a lab that publicly corrected its own contaminated score.

Move up only when you hit a specific wall: context (DeepSeek V4 Flash), licensing (Granite 4.2-8B), or ecosystem maturity (Qwen3.5-9B).

Last verified: September 7, 2026.