Overview with affiliate links. We receive a commission if you click through via us. Read how
Compare

What Is HBM Memory and Why Does It Matter for AI?

An explanation of HBM memory, how it differs from GDDR and why it plays such a big role in AI GPUs.

Blog6 min read

Short answer

HBM (high bandwidth memory) is memory that is placed in stacked layers right next to the GPU. This gives it a much higher bandwidth than ordinary graphics memory (GDDR). That matters for AI, because large models have to fetch a lot of data from memory. HBM is more complex to make and is mainly used in data centre GPUs.

HBM and GDDR compared

HBM and GDDR at a glance (general tendency)
AspectHBMGDDR
PlacementStacked, right next to the GPU chipSeparate chips around the GPU on the card
BandwidthVery highHigh
UseMainly data centre GPUsMainly graphics cards
ProductionMore complexSimpler

Why is bandwidth so important?

In AI inference, the GPU has to read the model’s weights from memory for every step. The speed at which that happens often determines how quickly a model responds. That is why AI GPU manufacturers put so much emphasis on their bandwidth. On our professional and data centre GPUs page you can see the bandwidth per product.

Why do you hear so much about it?

Because of the high demand for AI servers, memory is a bottleneck in the market. Read our post on the memory shortage for what sources report about it.

What does this mean for your choice?

For a GPU you mainly want to use for large language models, memory bandwidth is one of the first things to check. For a graphics card used for gaming or creative work, GDDR is usually more than enough. So look at the type of work you do.

Also pay attention to the total capacity: HBM memory is complex to make and is tied to the card. If your model does not fit, you cannot expand it. So add up the memory you need for model weights, intermediate results and input.

How do you calculate memory bandwidth from a spec sheet?

On a spec sheet, bandwidth is usually given as a final figure in GB/s, but you can also work it out yourself. That is useful if you want to check two figures or if only bus width and speed are given. The formula is simple:

bandwidth (GB/s) = bus width (bits) × data rate per pin (Gb/s) ÷ 8

Fill in the figures from your card’s spec sheet. You divide by 8 because a byte consists of eight bits. A very wide bus at a moderate speed can reach the same bandwidth as a narrow bus at a high speed. That is the idea behind HBM: a wide connection per stack.

JEDEC, the organisation that sets memory standards, describes an interface of 1024 bits per stack for HBM. For HBM3, JEDEC gives a data rate of up to 6.4 Gb/s per pin, good for 819 GB/s per memory device. That sum checks out: 1024 × 6.4 ÷ 8. A GPU often has several stacks, so the total on a card is higher. Which generation and speed a product uses is listed on the manufacturer’s page.

Is bandwidth enough, or does capacity also matter for your AI model?

You need both. Bandwidth determines how quickly data reaches the compute cores. Capacity determines whether the model and its intermediate results fit in at all.

Add up for yourself: weights, intermediate results and room for the system. That total must stay within the card’s memory. You can find more background in GPU specifications explained and Running AI at home or in the cloud.

What mistakes are easy to make when reading memory specifications?

Memory figures are easy to confuse. This table shows four common misunderstandings.

ConfusionWhat is going onHow to check
GB/s or Gb/sGigabyte and gigabit differ by a factor of 8Look at the capital B and divide gigabits by 8
Per pin or totalThe speed per pin is not the card’s bandwidthUse the formula above with the bus width
Maximum or real-worldStated bandwidth is a theoretical maximumLook for measurements under a load similar to yours
Per stack or per cardCapacity and bandwidth may be stated per componentLook for the total on the card’s product page

If in doubt, go by the manufacturer’s product page. On our GPU page and under compare, the data is shown with source references.

Which questions should you ask yourself before choosing a GPU for AI?

Go through these questions with your own figures:

  1. How big is the model? With N billion parameters and B bytes per parameter, you need roughly N × B gigabytes for the weights.
  2. How long are the conversations and how many people use it at the same time? That determines the room for intermediate results.
  3. Does the total fit in the memory of one card? If not, you need several cards or a different set-up, and the connection between the cards also comes into play. Read PCIe versions explained.
  4. Which memory type and bandwidth does the manufacturer state? Calculate them with the formula above.
  5. Does it also need to run at home? For irregular use, the cloud can be an alternative.

That gives you a well-founded list to go through the manufacturer’s specifications with.

Frequently asked questions

Is HBM memory for my PC?

No, HBM sits on the card itself and cannot be expanded separately.

Does a graphics card also have HBM?

That differs per product; most graphics cards use GDDR.

What is the difference between GB/s and Gb/s?

GB/s is gigabytes per second and Gb/s is gigabits per second. A byte consists of eight bits, so divide a figure in Gb/s by 8 to get GB/s. Check which unit is stated on the spec sheet.

Further reading

Last updated on

Ask us
Compare Get in touch

Are you sure?