What Is HBM? AI Memory Explained

NVIDIA's H200 pairs 141 GB of memory with 4.8 terabytes per second of bandwidth. That memory is HBM, and it exists because even a very fast AI chip produces little while it waits for data.

Watch it play outA fast chip does little while it waits for data

An AI chip can do math at extraordinary speed.

But data has to reach it first. Over a long, narrow path, the chip sits idle while it waits.

HBM stacks memory right beside the chip. The path gets short and wide, and the math units stay fed.

On NVIDIA's H200, that is 141 GB of HBM moving 4.8 terabytes per second.

Replay

High-bandwidth memory is DRAM (the standard working memory in computers) stacked into towers and placed right beside the AI chip. That short, wide path keeps the chip's math units fed, which is why memory has moved from a supporting part to a strategic one in the AI buildout. Further down, one line of arithmetic shows how directly memory caps a chip's speed.

Watch it play outHBM is ordinary DRAM piled into a tower beside the chip

Start with DRAM, the standard working memory in computers: thin slices of memory silicon.

HBM stacks those dies into a tower.

Then it places the tower right beside the AI chip, on the same package.

That short, wide path keeps the chip's math units fed. It is why memory became a strategic part of the AI buildout.

Replay

The short version

• HBM is one memory tier inside an AI accelerator, not a chip or a hard drive. Its job is moving huge amounts of data per second.
• An H200 carries 141 GB of HBM3e at 4.8 TB per second. Google's TPU v5p carries 95 GiB at 2,765 GB per second.

Watch it play outTwo accelerators, two very different data pipes

Each accelerator carries its own HBM, wired straight into the chip.

NVIDIA's H200: 141 GB of HBM3e streaming at 4.8 TB per second.

Google's TPU v5p: 95 GiB at 2,765 GB per second. Its dots travel slower, in proportion to the bandwidth.

Bandwidth, not just capacity, is HBM's job: how much data reaches the math units every second.

Replay

• Micron said in June 2026 that HBM4 was in high-volume shipments for its lead customer and that HBM4E volume production was expected in calendar 2027.
• The winning supplier won't automatically be the one announcing the most capacity. Qualification, yield and generation timing decide it.

Check the claimThree claims to test as you read

ClaimHBM is just faster laptop memory.Tap to checkPartly trueBoth are DRAM, but HBM stacks the dies beside the processor on the same package, buying much more bandwidth at a higher cost.Flip backClaimThe supplier announcing the most capacity wins.Tap to checkMythQualification, yield and generation timing decide it.Flip backClaimLower precision is good news for memory makers.Tap to checkMythStoring numbers in fewer bytes cuts the memory each task needs: 140 GB of weights become 70 GB.Flip back

What HBM actually is

HBM stands for high-bandwidth memory. Several DRAM dies (thin slices of memory silicon) are stacked on top of each other and connected to the accelerator package through a very wide interface. An accelerator is a chip built to speed up AI work, such as a GPU or Google's TPU.

An HBM stack is DRAM piled into a tower beside the chip

1. DRAM dieThin slice of memory silicon
2. DRAM die
3. DRAM dieSeveral dies, stacked and connected
4. Very wide interfaceInto the accelerator package

One HBM stack

AI performance depends on three separate memory traits:

• Capacity: how much data can sit close to the chip
• Bandwidth: how much data can move per second
• Latency: how long a single transfer takes

HBM is built for bandwidth and for moving data with less energy. To turn an accelerator count into installed memory and total bandwidth, try the free HBM demand calculator. It has sourced presets for the H100, H200, MI300X and MI325X, and every assumption is editable.

Why AI chips need it

AI models constantly shuttle large arrays of numbers through processors: model weights, activations (the intermediate outputs of each layer) and other working results. An accelerator can have extraordinary math throughput and still sit idle while that data arrives.

A fast chip produces little while it waits for data

1. HBM (Weights and activations) to Short, wide path (4.8 TB/s on an H200)
2. Short, wide path (4.8 TB/s on an H200) to Matrix units (Do the actual math)

Google's TPU architecture documentation describes parameters moving from HBM into the matrix-multiplication units that do the actual math. Here is how much memory two current chips carry:

Sources: NVIDIA and Google Cloud primary specifications.

Why ordinary server memory isn't enough

Conventional server memory still matters for CPUs, storage caches and large inference systems (inference means running a trained model to answer requests). But it connects through narrower channels outside the chip package, which limits how fast data reaches an accelerator.

HBM fixes part of that by sitting beside the processor on an advanced package. The price is manufacturing complexity and cost.

HBM buys bandwidth with manufacturing complexity

Conventional server memory

• Separate modules
• Narrower channels outside the chip package
• Still matters for CPUs and caches

Limits how fast data reaches an accelerator

vs

HBM

• Stacked DRAM dies
• Beside the processor on an advanced package
• Much more bandwidth, less energy per move

Higher cost, harder to make

Because HBM is scarce and expensive, chip designers still use caches, conventional DRAM, storage, compression, quantization (storing numbers with fewer bits) and model partitioning. The goal is to keep only the most urgent data closest to compute.

HBM3E, HBM4 and the naming ladder

Each generation tries to raise bandwidth, capacity, energy efficiency, or all three.

The name alone doesn't set performance. Stack height, package design, chip architecture, software, yield, power, cooling and interconnect all matter.

Micron said in June 2026 that HBM4 was in high-volume shipments for its lead customer's platform and that HBM4E volume production was expected in calendar 2027. See Micron's fiscal third-quarter 2026 results.

Spot the red flag

One phrase in Micron's update is a plan, not a shipment. Tap it.

Micron, June 2026

HBM4 was in high-volume shipments for its lead customer's platform, and HBM4E volume production was expected in calendar 2027.

Tap the phrase you think is the tell.

Spotted it.Not that phrase. Track each generation's milestone on its own: HBM4 ships today; HBM4E is a 2027 plan.

• “HBM4 was in high-volume shipments” A shipment milestone that has already happened.
• “for its lead customer's platform” The customer status: one lead customer.
• “and HBM4E volume production” The next product, not its status.
• “was expected in calendar 2027” Expected production is a plan. Nothing has shipped yet.

Explain every phraseHide the other explanations

Why supply is slow to grow

HBM eats more than DRAM wafer capacity. Stacked memory can need more wafer area per useful bit than ordinary memory, so shifting production toward HBM can tighten supply elsewhere in the memory market.

After that come the stacking, advanced packaging (fusing memory and chip into one unit), testing and customer qualification. A problem at any stage cuts usable output. Our HBM yield and supply-chain guide walks through each stage and the milestones that separate real shipments from announcements.

A problem at any stage cuts usable output

1. DRAM wafers (More area per useful bit) to Stack and package (Fuse memory and chip)
2. Stack and package (Fuse memory and chip) to Test
3. Test to Customer qualification (Usable output)

How bandwidth changes AI economics

A chip that waits on memory still burns capital, power, cooling and floor space. More bandwidth can raise useful output without buying a matching amount of extra compute.

The effect depends on the job:

• Large-model inference can be limited by moving model weights and keeping a key-value cache (the model's short-term memory of a conversation).
• Training can be limited by memory capacity, data movement or communication across many chips.
• Recommendation systems can need very large embedding tables.
• Smaller or heavily optimized models may be limited by compute instead.

"More HBM equals faster" is not a universal rule. The bottleneck has to be measured for each system and workload.

Member research: A chip stalled on memory still has to be paid for, and that cost reaches earnings as depreciation. Issue 03 shows four hyperscalers put almost $4 into Q1 infrastructure for every $1 of depreciation, and where the margin test lands next.

Worked example: does a model fit on one H200?

Hypothetical example: a model with 70 billion parameters (the learned numbers that make up the model) runs on one NVIDIA H200, using the 141 GB and 4.8 TB per second from the spec sheet. The memory a model's weights need is simply parameters times bytes per parameter.

Make a guess

A 70-billion-parameter model stored at 16-bit precision (2 bytes each) runs on one H200 with 141 GB of HBM. How much memory is left for conversations?

Tap your guess

Under 10 GBYour guess10–40 GBYour guess40–70 GBYour guessOver 70 GBYour guess

0123456789 GBActual

Spot on.Close, a little low.Close, a little high.Way off, too low.Way off, too high. It was 1 GB.

The weights alone take 140 GB, so in practice you'd need a second chip, which means buying more HBM.

Actual1 GB

Left at 8-bit precision71 GB

Two lessons fall out:

• Capacity. At 16-bit precision the weights alone nearly fill the chip, leaving almost no room for conversations. In practice you'd need a second chip, which means buying more HBM.
• Bandwidth. When a model writes text one token at a time for a single user, it typically reads every weight once per token. So bandwidth sets a rough speed ceiling of about 34 tokens a second at 16-bit, and about 69 at 8-bit. Serving many users in one batch shares each read, which is why providers batch requests.

Halving precision doubles the speed ceiling

16-bit precision34 tokens/s

8-bit precision69 tokens/s

Single user, one full read of the weights per token: 4,800 GB/s ÷ weight size. A ceiling, not a benchmark.

Halving the precision doubled the ceiling and freed half the memory without adding any compute. That is the upside of quantization for AI companies, and the reason it belongs on a memory investor's risk list. The example ignores cache reads, software overhead and multi-chip setups, so treat it as a ceiling, not a benchmark.

Who earns money along the chain

A hyperscaler is one of the giant cloud platforms. HBM belongs inside the wider 2026 hyperscaler capex cycle (capex being the money spent on long-lived assets like data centers), not in a standalone commodity story.

Try it yourself

A 10-minute exercise with the free tools:

1. Open the HBM demand calculator and choose the NVIDIA H200 preset.
2. Enter 70 billion parameters at FP16 / BF16 (2 bytes) with one concurrent sequence. Check whether it fits, and compare the result with the worked example above.
3. Switch precision to FP8 / INT8, then raise the context and concurrent sequences until you run out of memory. Watch how fast the key-value cache grows.
4. Switch to the AMD MI325X preset and see how many accelerators the same deployment needs.
5. Check the H200 preset against NVIDIA's own spec page.

You should now be able to answer: for this workload, is the limit memory capacity, bandwidth or neither, and how much would lower precision cut the HBM it needs?

Our read

HBM matters because AI economics depend on moving data, not just doing arithmetic. That makes it a real bottleneck, but a bottleneck in a cyclical industry with a long history of boom and bust.

What we'd watch:

• Generation transitions. A supplier can lead one generation and lag the next.
• Gross margin. Scarcity produces exceptional margins, and those margins attract new supply that eventually brings them down.
• Customer concentration. Demand rests on a small number of chip platforms and hyperscalers, so one delayed product can hit several suppliers at once.
• Efficiency gains. Quantization, sparsity, caching and better scheduling reduce memory per task. Total demand can still rise, but unit economics may improve faster than expected.
• Competing designs. Larger on-chip cache, conventional DRAM, lower-power memory, pooled memory and workload-specific chips can change the ideal mix.

The case weakens if margins fall while capacity keeps rising, or if efficiency gains start shrinking memory per task faster than usage grows. It strengthens if suppliers hold margin through the HBM4 transition. For the next layer of the stack, read custom AI chips versus GPUs.

What would change our read on HBM

Case strengthens

• Suppliers hold margin through the HBM4 transition

vs

Case weakens

• Margins fall while capacity keeps rising
• Efficiency shrinks memory per task faster than usage grows

Questions readers ask

Is HBM the same as the memory in my laptop?

Both are DRAM. Ordinary memory sits on separate modules and connects through narrower channels. HBM stacks the dies and sits beside the processor on the same package, which buys much more bandwidth at a higher cost and with harder manufacturing.

Does more HBM always make AI faster?

No. Large-model inference and some training jobs are memory-limited, but smaller or heavily optimized models can be limited by compute instead. The worked example shows that capacity and bandwidth are separate limits, and a chip can hit either one first.

Why would lower precision hurt memory makers?

Storing each parameter in fewer bytes shrinks the model. In the example, going from 16-bit to 8-bit cut the weights from 140 GB to 70 GB. Total HBM demand can still rise if usage grows fast enough, but each task needs less memory.

How can I tell if a memory maker is winning in HBM?

Look past capacity announcements. Qualified customers, volume shipments, yield, gross margin and generation timing tell you more. The HBM yield guide explains how to separate samples from real shipments.

Go deeper with member research

Understanding HBM is step one. These member case studies ask what it was worth to shareholders, and when:

• The HBM Capacity Trade: Micron’s HBM capacity thesis as the evidence stood on December 18, 2025, with timing, valuation and later results kept separate.
• Lam Research and the economics of harder-to-build memory: NAND conversions, manufacturing complexity and cash conversion as of January 30, 2026.

Limits of this data

• Product specifications and production plans change. Check company statements against current filings and technical documentation.
• The specs above are vendor figures, not measured performance on any particular workload. The worked example is a simplified ceiling, not a benchmark.