What is the difference between a MoE and a dense model?

A dense model runs every parameter for every token. A mixture-of-experts model splits its feed-forward layers into many expert sub-networks and a router sends each token to only a few of them, so most parameters stay idle per token. GLM-5.3-Flash has 320 billion total parameters but activates only 18 billion per token; Muse Glimmer 30B is dense and activates all 30 billion.

What do active parameters actually predict?

Active parameters predict compute cost and speed: how much arithmetic runs per token, which drives latency and API price. Total parameters predict memory: every parameter must sit in GPU RAM whether or not it fires. That is why a 125B model with 6B active is fast but still needs roughly 70-80GB at 4-bit precision to run at all.

Is MoE better than dense?

Neither is better in general. MoE gives more capability per unit of compute, which wins for API serving where the provider absorbs the memory cost. Dense gives more capability per unit of memory, which wins for local and edge deployment where memory is the binding constraint. Choose by which resource you are short of.

How do I estimate memory for an MoE model?

Use total parameters, not active. Roughly, bytes needed equals total parameters times bytes per parameter: 2 bytes at BF16, about 0.5 bytes at 4-bit quantisation. Then add 20-40% for KV cache, activations, and framework overhead, and more if you plan to use a long context window. A 320B model at 4-bit lands near 192GB in practice.

Quick Answer

MoE vs Dense LLMs Explained: What Active Params Mean

Published: August 27, 2026

The Short Answer

A dense model runs every parameter for every token. A mixture-of-experts (MoE) model splits its feed-forward layers into many expert sub-networks, and a router sends each token through only a few of them — so most parameters sit idle on any given token.

That is why 2026 model cards quote two numbers, and why only one of them predicts your bill:

Active parameters predict compute — speed and price. Total parameters predict memory — whether it runs at all.

Almost every expensive mistake in local AI deployment comes from reading the active number and budgeting hardware against it.

The Two Numbers, With Real Models

Verified against current model cards, August 27, 2026:

Model	Total	Active	% active	Type
Qwen3.8-Flash-Next	125B	6B	4.8%	MoE
GLM-5.3-Flash	320B	18B	5.6%	MoE
Muse Glimmer 30B	30B	30B	100%	Dense
Qwen3.8-27B	27B	27B	100%	Dense

Qwen3.8-Flash-Next does the arithmetic of a 6B model and needs the memory of a 125B one. Read that sentence twice before buying a GPU.

How Routing Actually Works

Inside each MoE layer sits a small router — a learned network that scores every expert for the current token and picks the top-k, typically two.

Concretely, for a model with 512 experts and top-2 routing: the router evaluates 512 cheap scores, selects 2 experts, runs the token through those 2, and combines the outputs weighted by the router’s confidence. The other 510 experts contribute nothing to that token — but they still occupy GPU memory, because the next token may route to them.

Experts do not partition neatly by topic. Training discovers the specialisation, and it usually looks like nothing a human would name — syntax patterns, numeric contexts, specific token neighbourhoods. Do not reason about MoE as “the code expert and the French expert.” That intuition is wrong and it will mislead you.

Two consequences worth knowing:

Load balancing is a real training problem. Left alone, routers collapse onto a favourite handful of experts and the rest never learn. Auxiliary losses exist specifically to prevent this, and they are a live source of training instability.

Batching gets harder. Different tokens in the same batch route to different experts, so the hardware juggles many small matrix multiplications instead of one big one. This is why MoE inference engines are more complex than dense ones, and why naive MoE serving can underperform its FLOP count.

Estimating Memory — The Calculation That Matters

Use total parameters.

memory ≈ total_params × bytes_per_param × overhead

Precision	Bytes/param	30B model	125B model	320B model
BF16	2	~60GB	~250GB	~640GB
8-bit	1	~30GB	~125GB	~320GB
4-bit	~0.5	~15GB	~63GB	~160GB

Then add 20-40% overhead for KV cache, activations and framework slack — more if you intend to use a long context window, since KV cache grows with context length and can dominate at 1M tokens.

That overhead is why GLM-5.3-Flash’s practical 4-bit requirement is quoted around 192GB rather than the naive 160GB, and why Qwen3.8-Flash-Next lands near 70-80GB rather than 63GB.

The single-GPU test: can your card hold total_params × 0.5 bytes × 1.3? If not, the active-parameter count is irrelevant to you.

When Each Wins

MoE wins when memory is cheap and compute is expensive. That is the API provider’s world. A datacenter can afford 192GB of HBM; what it cannot afford is burning frontier-scale FLOPS on every token of a summarisation job. This is precisely why the cheapest capable APIs in 2026 are sparse MoE — GLM-5.3-Flash serves a 320B-class model at $0.15/$0.50 per MTok because it only pays compute for 18B.

Dense wins when memory is the binding constraint. That is your laptop, your workstation, your edge device. Meta built Muse Glimmer 30B dense and Apache 2.0 specifically so it fits one consumer GPU — a deliberate rejection of sparsity for a deployment target where sparsity does not help.

	MoE	Dense
Capability per FLOP	✅ High	Lower
Capability per GB	Lower	✅ High
Serving complexity	Higher	✅ Simple
Fine-tuning ease	Harder	✅ Easier
Best for	API / datacenter	Local / edge

Fine-tuning is the underrated dense advantage. MoE fine-tuning has to contend with routing collapse — narrow task data pushes the router toward a few experts and quietly destroys general capability. Dense models have no router to break. If you plan to fine-tune on a modest dataset, dense is the lower-risk choice.

The Direction of Travel

Sparsity ratios are getting more aggressive. Qwen3.8-Flash-Next activates 4.8% of its network per token — and Qwen released it explicitly as a preview of the Qwen4 architecture, which is a public commitment to that direction.

The accompanying work is in attention, not just routing. Qwen’s QSA selects at micro-block rather than token level to cut long-context latency, because agentic workloads accumulate context across many steps and latency compounds per step.

What this means practically: the gap between “what an API can serve cheaply” and “what you can run at home” is widening, not narrowing. Sparse MoE makes cloud inference dramatically cheaper without making local inference any easier. Plan your architecture around that split rather than hoping it closes.

The Practical Rule

Choosing an API? Ignore both parameter counts. Compare cost per task and benchmark quality. Sparsity is the provider’s problem.
Running locally? Filter on total parameters first, then compare quality among whatever fits. Active parameters only tell you how fast the survivors will be.
Fine-tuning on a small dataset? Prefer dense.
Never budget hardware against an active-parameter count. It is the most common and most costly mistake in this space.