What is VRAM and how much GPU memory do you need?
Explanation of VRAM, what it determines and how to estimate how much GPU memory your AI work needs.
VRAM is a GPU’s own working memory. It determines how large an AI model or dataset can be that you load in one go. How much you need depends on the size of the model, the precision it computes at and whether you train or just use it. Training needs far more than inference.
Who is it for: Anyone who wants to rent a GPU in the cloud for AI and wants to know where the memory comes in.
What is VRAM?
VRAM stands for video random access memory. It is memory that sits directly on the GPU and is fast. A GPU can only compute with data that fits in that memory. If a model does not fit, you have to split it across several GPUs, make it smaller or take a different card.
Where does the memory go?
The memory is used for the model’s weights, for intermediate results during computation and, in training, also for loss calculations and optimisation data. Training therefore needs a multiple of what inference needs. With long inputs, memory use grows too.
How do you estimate it?
Start with the size of the model, usually expressed in the number of parameters, and the precision you compute at. The lower the precision, the less memory per parameter. Add space for intermediate results. The documentation of the model or the software often gives a guideline. Test with a small example to see what you actually use.
If it does not fit
You have a few options: a smaller or compressed model, a card with more memory, splitting the model across several GPUs or a smaller batch size. Each choice has consequences for speed, costs and complexity.
Memory and storage are not the same
VRAM is the GPU’s working memory. Storage, such as object storage or a disk, keeps your data and models in the long term. Make sure the storage delivers data fast enough, otherwise the GPU waits. See the guide on data storage for AI.
Memory in AI work
| Situation | Memory need | Explanation |
|---|---|---|
| Using a small model | Low to moderate | Often fits on one card |
| Using a large model | High | May require several cards |
| Fine-tuning a model | Moderate to high | Depends on the chosen method |
| Training a model | Very high | Usually several GPUs |
| Long input | Extra | Memory grows with the length |
Who is it not ideal for?
- Anyone who only uses an online AI service and hosts nothing themselves.
Frequently asked questions
Is more VRAM better in every case?
Only if you need it. If your model already fits, extra memory does not necessarily make computing faster.
Is VRAM different from ordinary working memory?
Yes. VRAM sits with the GPU, the working memory with the CPU.
Can I expand VRAM later?
Not on an existing card. In the cloud you switch instance or add GPUs.
