Overview with affiliate links. We receive a commission if you click through via us. Read how
Compare

What is VRAM and how much GPU memory do you need?

Explanation of VRAM, what it determines and how to estimate how much GPU memory your AI work needs.

Door de redactie van ClouwinoGepubliceerd Laatst bijgewerkt

Short answer

VRAM is a GPU’s own working memory. It determines how large an AI model or dataset can be that you load in one go. How much you need depends on the size of the model, the precision it computes at and whether you train or just use it. Training needs far more than inference.

Who is it for: Anyone who wants to rent a GPU in the cloud for AI and wants to know where the memory comes in.

What is VRAM?

VRAM stands for video random access memory. It is memory that sits directly on the GPU and is fast. A GPU can only compute with data that fits in that memory. If a model does not fit, you have to split it across several GPUs, make it smaller or take a different card.

Where does the memory go?

The memory is used for the model’s weights, for intermediate results during computation and, in training, also for loss calculations and optimisation data. Training therefore needs a multiple of what inference needs. With long inputs, memory use grows too.

How do you estimate it?

Start with the size of the model, usually expressed in the number of parameters, and the precision you compute at. The lower the precision, the less memory per parameter. Add space for intermediate results. The documentation of the model or the software often gives a guideline. Test with a small example to see what you actually use.

If it does not fit

You have a few options: a smaller or compressed model, a card with more memory, splitting the model across several GPUs or a smaller batch size. Each choice has consequences for speed, costs and complexity.

Memory and storage are not the same

VRAM is the GPU’s working memory. Storage, such as object storage or a disk, keeps your data and models in the long term. Make sure the storage delivers data fast enough, otherwise the GPU waits. See the guide on data storage for AI.

Memory in AI work

A rough indication; actual requirements depend on the model and software.
SituationMemory needExplanation
Using a small modelLow to moderateOften fits on one card
Using a large modelHighMay require several cards
Fine-tuning a modelModerate to highDepends on the chosen method
Training a modelVery highUsually several GPUs
Long inputExtraMemory grows with the length

Who is it not ideal for?

Frequently asked questions

Is more VRAM better in every case?

Only if you need it. If your model already fits, extra memory does not necessarily make computing faster.

Is VRAM different from ordinary working memory?

Yes. VRAM sits with the GPU, the working memory with the CPU.

Can I expand VRAM later?

Not on an existing card. In the cloud you switch instance or add GPUs.

Ask us
Compare Get in touch

Are you sure?