What is the difference between HBM and GDDR7?

HBM goes wide and slow, GDDR7 goes narrow and fast. HBM stacks DRAM dies vertically next to the processor and uses a very wide interface — 1,024 data I/Os per stack through HBM3E, doubling to 2,048 in HBM4. GDDR7 uses a much narrower bus at far higher per-pin transfer rates. HBM wins on bandwidth per watt and total bandwidth; GDDR7 wins on cost and manufacturing simplicity, which is why it dominates consumer GPUs.

How much faster is HBM4 than HBM3E?

The most defensible figure is Micron's own published comparison — about 2.3 times the bandwidth of an equivalent-capacity HBM3E part with more than 20% better power efficiency. Absolute per-stack numbers quoted in the press range from roughly 2 TB/s to 3.3 TB/s depending on the vendor and bin, so treat specific TB/s claims with caution and check the vendor datasheet.

Why does memory bandwidth matter more than FLOPS for LLM inference?

Autoregressive token generation is memory-bound, not compute-bound. Producing each token requires streaming the model's weights from memory, so throughput is governed by how fast weights can be read rather than how many operations the chip can perform. This is why an accelerator with more HBM bandwidth often beats one with higher peak FLOPS on real inference workloads.

Should I care about HBM when renting GPUs?

Yes, for inference sizing. Total HBM capacity determines which models fit without splitting across devices, and HBM bandwidth largely determines your tokens per second. When comparing rental options, check memory capacity and bandwidth before peak FLOPS — capacity decides feasibility and bandwidth decides speed for most LLM serving.

Quick Answer

HBM4 vs HBM3E vs GDDR7: AI Memory Explained (2026)

Published: August 28, 2026

The Short Answer

HBM goes wide and slow. GDDR7 goes narrow and fast.

That single architectural choice explains everything downstream — why HBM costs more, why it wins on power efficiency, why it needs advanced packaging, and why every serious AI accelerator uses it while every gaming GPU does not.

For LLM inference specifically, memory bandwidth is usually the binding constraint, not compute. If you optimise for FLOPS and ignore bandwidth, you will buy the wrong hardware.

The Comparison

	HBM3E	HBM4	GDDR7
Interface width	1,024 I/Os per stack	2,048 I/Os per stack	narrow bus (e.g. 384-bit)
Strategy	very wide, moderate speed	wider still, logic base die	narrow, very high per-pin rate
Bandwidth	~1.2 TB/s per stack	~2.3× HBM3E at equal capacity	far lower per device
Power efficiency	good	>20% better than HBM3E	worse per bit moved
Packaging	advanced, stacked + TSV	advanced, stacked + TSV	conventional, on-board
Cost	high	higher	low
Supply	constrained	constrained	ample
Found in	AI accelerators	next-gen AI accelerators	consumer/pro GPUs

The 2.3× figure and the >20% power number are Micron’s own published apples-to-apples comparison at held-constant capacity, which makes them the most conservative claims available. Absolute per-stack bandwidth claims vary widely across secondary sources — roughly 2 TB/s to 3.3 TB/s — because they mix vendors, bins and generations. Check the datasheet for the specific part.

Why Wide Beats Fast for AI

Moving a bit costs energy roughly in proportion to how fast and how far you push it. Two ways to get bandwidth:

Narrow and fast (GDDR7): push each pin to extreme transfer rates. Energy per bit rises steeply. Signal integrity gets hard. But the memory sits on a normal PCB and manufacturing is conventional and cheap.

Wide and moderate (HBM): run thousands of parallel lanes at sane speeds. Energy per bit stays low. But the lanes only work over millimetres, so the memory must be stacked and bonded right next to the processor — which requires advanced packaging, and that is precisely the industry’s supply bottleneck.

HBM4’s headline change is doubling the interface from 1,024 to 2,048 I/Os per stack, plus a logic base die. It is the same bet, doubled.

Why This Decides Your Inference Speed

Generating tokens autoregressively means: for every single token, stream the model weights out of memory.

A rough mental model for a memory-bound decode:

tokens_per_second ≈ memory_bandwidth ÷ model_bytes_in_memory

A 70B model at 8-bit occupies roughly 70GB. On an accelerator with ~3 TB/s of usable bandwidth, the ceiling is on the order of ~40 tokens/second per sequence — regardless of the chip’s peak FLOPS. Batching amortises this across sequences, which is exactly why serving stacks batch aggressively, but the single-stream ceiling is set by bandwidth.

This is why memory capacity and bandwidth belong at the top of your hardware checklist:

Capacity decides whether the model fits at all. Size on total parameters, not active parameters — a sparse MoE with 18B active still needs all its weights resident.
Bandwidth decides how fast it runs.
FLOPS matters mostly for prefill and training, not decode.

Practical Guidance

Renting GPUs for inference? Compare memory capacity first (does the model fit on one device?), bandwidth second (how many tokens/sec?), FLOPS third. A cheaper card that forces multi-GPU sharding for a model that would fit on one bigger card is usually a false economy once you count interconnect overhead and complexity.

Running local models? Unified-memory systems trade absolute bandwidth for large capacity at low cost. That is a genuinely good trade for large-model experimentation and a bad one for latency-sensitive serving.

Buying consumer GPUs for LLM work? GDDR7 devices are bandwidth-limited relative to HBM parts, and capacity is usually the harder wall — most consumer cards cannot hold a large model at all. Quantisation is the lever that makes this workable, at a quality cost you should measure rather than assume.

The Supply Angle

HBM’s dependence on advanced packaging is not a footnote — it is the reason accelerator supply has repeatedly been gated on memory rather than logic dies. That is also why capacity investments target packaging specifically: SK hynix broke ground on a $4 billion-plus advanced packaging plant in West Lafayette, Indiana on August 27, 2026, its first HBM base in the US, with the cleanroom targeted for October 2028.

The practical implication for buyers: HBM-equipped capacity stays constrained through at least 2028. Plan procurement accordingly, and do not assume next year’s accelerator will be easy to get just because it has been announced.