What Is HBM Memory and Why Does It Matter for AI?
An explanation of HBM memory, how it differs from GDDR and why it plays such a big role in AI GPUs.
HBM (high bandwidth memory) is memory that is placed in stacked layers right next to the GPU. This gives it a much higher bandwidth than ordinary graphics memory (GDDR). That matters for AI, because large models have to fetch a lot of data from memory. HBM is more complex to make and is mainly used in data centre GPUs.
HBM and GDDR compared
| Aspect | HBM | GDDR |
|---|---|---|
| Placement | Stacked, right next to the GPU chip | Separate chips around the GPU on the card |
| Bandwidth | Very high | High |
| Use | Mainly data centre GPUs | Mainly graphics cards |
| Production | More complex | Simpler |
Why is bandwidth so important?
In AI inference, the GPU has to read the model’s weights from memory for every step. The speed at which that happens often determines how quickly a model responds. That is why AI GPU manufacturers put so much emphasis on their bandwidth. On our professional and data centre GPUs page you can see the bandwidth per product.
Why do you hear so much about it?
Because of the high demand for AI servers, memory is a bottleneck in the market. Read our post on the memory shortage for what sources report about it.
What does this mean for your choice?
For a GPU you mainly want to use for large language models, memory bandwidth is one of the first things to check. For a graphics card used for gaming or creative work, GDDR is usually more than enough. So look at the type of work you do.
Also pay attention to the total capacity: HBM memory is complex to make and is tied to the card. If your model does not fit, you cannot expand it. So add up the memory you need for model weights, intermediate results and input.
How do you calculate memory bandwidth from a spec sheet?
On a spec sheet, bandwidth is usually given as a final figure in GB/s, but you can also work it out yourself. That is useful if you want to check two figures or if only bus width and speed are given. The formula is simple:
bandwidth (GB/s) = bus width (bits) × data rate per pin (Gb/s) ÷ 8
Fill in the figures from your card’s spec sheet. You divide by 8 because a byte consists of eight bits. A very wide bus at a moderate speed can reach the same bandwidth as a narrow bus at a high speed. That is the idea behind HBM: a wide connection per stack.
JEDEC, the organisation that sets memory standards, describes an interface of 1024 bits per stack for HBM. For HBM3, JEDEC gives a data rate of up to 6.4 Gb/s per pin, good for 819 GB/s per memory device. That sum checks out: 1024 × 6.4 ÷ 8. A GPU often has several stacks, so the total on a card is higher. Which generation and speed a product uses is listed on the manufacturer’s page.
Is bandwidth enough, or does capacity also matter for your AI model?
You need both. Bandwidth determines how quickly data reaches the compute cores. Capacity determines whether the model and its intermediate results fit in at all.
- Model weights: NVIDIA gives the example of a model with 7 billion parameters in 16-bit precision, which takes up roughly 14 GB of memory. The rule of thumb is: number of parameters × bytes per parameter.
- Intermediate results (KV cache): according to NVIDIA, this grows linearly with the length of the input and with the number of simultaneous requests. A long conversation or more users therefore costs extra memory on top of the weights.
- Speed when generating: NVIDIA describes generating text token by token as memory-bound. Transfer from memory then weighs more heavily than compute speed.
Add up for yourself: weights, intermediate results and room for the system. That total must stay within the card’s memory. You can find more background in GPU specifications explained and Running AI at home or in the cloud.
What mistakes are easy to make when reading memory specifications?
Memory figures are easy to confuse. This table shows four common misunderstandings.
| Confusion | What is going on | How to check |
|---|---|---|
| GB/s or Gb/s | Gigabyte and gigabit differ by a factor of 8 | Look at the capital B and divide gigabits by 8 |
| Per pin or total | The speed per pin is not the card’s bandwidth | Use the formula above with the bus width |
| Maximum or real-world | Stated bandwidth is a theoretical maximum | Look for measurements under a load similar to yours |
| Per stack or per card | Capacity and bandwidth may be stated per component | Look for the total on the card’s product page |
If in doubt, go by the manufacturer’s product page. On our GPU page and under compare, the data is shown with source references.
Which questions should you ask yourself before choosing a GPU for AI?
Go through these questions with your own figures:
- How big is the model? With N billion parameters and B bytes per parameter, you need roughly N × B gigabytes for the weights.
- How long are the conversations and how many people use it at the same time? That determines the room for intermediate results.
- Does the total fit in the memory of one card? If not, you need several cards or a different set-up, and the connection between the cards also comes into play. Read PCIe versions explained.
- Which memory type and bandwidth does the manufacturer state? Calculate them with the formula above.
- Does it also need to run at home? For irregular use, the cloud can be an alternative.
That gives you a well-founded list to go through the manufacturer’s specifications with.
Frequently asked questions
Is HBM memory for my PC?
No, HBM sits on the card itself and cannot be expanded separately.
Does a graphics card also have HBM?
That differs per product; most graphics cards use GDDR.
What is the difference between GB/s and Gb/s?
GB/s is gigabytes per second and Gb/s is gigabits per second. A byte consists of eight bits, so divide a figure in Gb/s by 8 to get GB/s. Check which unit is stated on the spec sheet.
Further reading
Last updated on
