How Does HBM Work? The Stacked Memory Powering AI Chips
Related video: "What is High-Bandwidth Memory (HBM)? HBM vs. GDDR" (via Eye on Tech)
The short answer
HBM works by turning memory from a flat layout into a vertical one. Instead of memory chips sitting on a stick in a slot several centimeters from the processor, eight, twelve, or sixteen memory chips are stacked on top of each other, drilled through with thousands of microscopic vertical wires, and mounted on the same package as the processor — millimeters away.
That physical rearrangement is the whole trick. It lets engineers run roughly 1,024 parallel data connections into a single stack, where a standard memory module gets about 64. Think of it as replacing a fast two-lane highway with a slower thousand-lane one. Total throughput goes up enormously.
Everything else about HBM — the eye-watering price, the Korean companies dominating it, the packaging bottlenecks — flows from that one design decision.
Why memory became the problem in the first place
For decades, processors got faster much more quickly than memory got faster at delivering data. Engineers call this the memory wall. A chip can only compute as fast as data arrives, and increasingly the data was the slow part.
Standard DRAM — the memory type in every PC, phone, and server — attacked this by making each wire run faster. DDR5 modules push very high per-pin speeds, and GDDR, the variant soldered around gaming graphics cards, pushes them harder still.
HBM went the opposite direction. Per wire, HBM is generally slower than DDR5 or GDDR. It wins on width. Bandwidth is roughly interface width multiplied by speed per wire, and HBM maximizes the first number instead of the second.
Here's the line worth remembering: HBM is not "faster memory." It's "wider memory."
So why can't you just build a 1,024-bit memory stick?
Because 1,024 wires have to physically exist somewhere. On a motherboard, every wire is a copper trace several centimeters long. A thousand of them wouldn't fit, would consume a great deal of power, and would degrade the signal along the way.
The only practical way to get that many connections is to make the distance almost zero. Hence stacking, and hence putting the stack on the same package as the processor. Stacking isn't mainly about saving space — it's what makes an ultra-wide interface physically possible at all.
Three terms that explain the stack
- TSV (Through-Silicon Via): microscopic holes etched straight down through each silicon chip and filled with metal, so signals travel vertically through the stack instead of around the edges. Picture an elevator shaft running through every floor of a building rather than external fire escapes. This is the defining technology of HBM.
- Base die (logic die): the bottom layer of the stack isn't memory at all. It's a traffic controller that talks to the processor. In newer HBM generations this base layer is becoming a genuinely custom logic chip, which is why chip foundries — companies that manufacture chips for others — are now involved in what used to be purely a memory business. From the HBM4 generation onward the base die is built on an advanced foundry logic process rather than a memory process: Samsung makes it on its own 4nm foundry node and is the only HBM4 supplier running memory, foundry, and packaging as a single in-house turnkey operation, while SK hynix — which has no foundry of its own — outsources the die to TSMC's 12nm process, is weighing a shift to TSMC 3nm from HBM4E, and is also looking at Intel Foundry for supply-chain diversification.
- Interposer (2.5D packaging): the HBM stacks don't sit on the motherboard. They sit shoulder-to-shoulder with the GPU on a thin slab of silicon, with the thousands of wires etched into that slab. The best-known industry version of this is TSMC's CoWoS (Chip-on-Wafer-on-Substrate).
Stack heights are described as 8-Hi, 12-Hi, or 16-Hi — meaning eight, twelve, or sixteen memory chips in one tower. More floors, more capacity per stack.
How the floors get joined together
The layers are connected by tiny solder bumps called microbumps, with the gaps filled by protective material. There are two competing approaches here, and this is where the Korean rivalry actually lives.
SK hynix is associated with a method called MR-MUF (mass reflow with molded underfill), while Samsung has leaned on TC-NCF (thermo-compression with non-conductive film). The trade-offs largely come down to heat dissipation and how thin each layer can be squeezed. Longer term, the industry is working toward hybrid bonding — joining copper directly with no bumps at all — to fit more layers into the same height, though that remains more roadmap than mainstream production.
HBM vs. regular memory: the honest comparison
HBM vs. Traditional Memory at a Glance
| Regular DRAM (DDR5) | GDDR (gaming GPUs) | HBM | |
|---|---|---|---|
| Physical form | Flat stick in a slot | Flat chips around the GPU | Vertical stack on the package |
| Interface width | ~64 bits per module | ~32 bits per chip | ~1,024 bits per stack |
| Speed per wire | High | Highest | Lowest of the three |
| Distance to processor | Centimeters | Centimeters | Millimeters |
| Energy per bit moved | Highest | High | Lowest |
| Cost per gigabyte | Lowest | Middle | Highest by a wide margin |
| User-upgradeable | Yes | No | No — it's sealed into the package |
Two caveats that most beginner explainers skip. First, HBM barely improves latency. The time for the first byte to show up is broadly comparable to normal DRAM. It's a wider pipe, not a shorter one.
Second, HBM is not replacing the memory in your laptop. You'd lose upgradeability and pay many times more per gigabyte for bandwidth ordinary software would never use.
Why AI is the reason anyone cares
A large language model is, physically, a gigantic pile of numbers called weights sitting in memory. To produce a single word of output, the chip has to read a large fraction of those numbers. Then it does it again for the next word. And again.
The arithmetic is comparatively easy for a modern GPU. The reading is the hard part. So for much of its runtime, AI inference is memory-bandwidth-bound rather than compute-bound. In those workloads, increasing memory bandwidth can translate almost directly into more words per second.
Capacity matters just as much. If the model's weights don't fit in the on-package memory, the system has to shuttle data in from slower memory, and performance drops sharply. HBM capacity effectively decides how large a model one accelerator can hold.
Then there's power. Moving a bit across a motherboard costs considerably more energy than moving it a few millimeters. At data-center scale, that energy advantage translates into how many chips you can fit inside a rack's power budget. For a sense of scale: a high-end desktop PC's memory bandwidth is typically measured in tens of gigabytes per second, while a single AI accelerator is measured in thousands.
Why it's so expensive — five structural reasons
This is the part that explains the Korean semiconductor boom better than any stock chart.
- You're buying many chips, not one. A 12-high stack is twelve memory chips plus a base die. Each of those chips also gives up silicon area to TSVs and test circuitry that normal DRAM doesn't need, so each one yields less usable memory.
- Yield multiplies against you. If one layer in a twelve-layer stack fails after bonding, the entire stack is scrap — including eleven perfectly good chips. Even a 99% success rate per layer compounds to roughly 89% across twelve. That's why "Known Good Die" testing, checking each chip before stacking, is such an obsession in this industry.
- Packaging is a separate scarce resource. HBM is useless without advanced 2.5D packaging capacity. Even when memory makers churn out stacks, finished AI chips are gated by packaging lines. Two bottlenecks in series, not one.
- Almost no competition. Three companies make HBM at scale: SK hynix, Samsung, and Micron. Three suppliers, explosive demand, and lengthy customer qualification cycles add up to real pricing power.
- It's largely sold before it's made. HBM capacity tends to be contracted well in advance, so it behaves less like a spot-priced commodity chip and more like a negotiated custom component.
What this means if you're watching Korea
South Korea has been a memory-chip superpower for decades, but memory was historically a brutal commodity business — boom, glut, price crash, repeat. HBM changed the character of that business. It's a differentiated, contracted, engineering-intensive product where stacking know-how and thermal management genuinely separate the winners from the losers.
That's why SK hynix, long the smaller sibling to Samsung in Korean semiconductors, has spent the AI boom in an unusually strong position. It's also why Samsung's HBM qualification progress with major AI chip customers has been covered in Korea as national news.
If you want one lens for understanding the Korean tech economy right now, it's this: the country's biggest export industry has shifted from selling interchangeable chips to selling very hard-to-build towers.
FAQ
Is HBM just faster RAM?
No. Per wire, HBM is generally slower than DDR5 or GDDR memory. It achieves higher total bandwidth by using far more parallel connections — roughly 1,024 per stack versus about 64 for a standard memory module. It's wider, not faster, and it doesn't meaningfully reduce latency.
Can I buy HBM for my PC?
Not as an upgrade. HBM is bonded onto the same package as the processor during manufacturing, so it can't be added, swapped, or expanded. A handful of past graphics cards shipped with it, but for consumer use it costs far more per gigabyte than GDDR while delivering bandwidth ordinary software can't exploit.
Which companies make HBM?
Only three at meaningful scale: SK hynix and Samsung Electronics of South Korea, and Micron of the United States. The standard itself is set by JEDEC, an industry body, but the manufacturing know-how — especially stacking and bonding — is concentrated in very few hands, which is a big part of why HBM commands such high prices.
Comments
Post a Comment