Running AI: At Home on Your Own GPU or in the Cloud?
The trade-offs between running AI on your own GPU and renting GPU capacity in the cloud.
Your own graphics card is suitable for experimenting with smaller models and for keeping data in your own hands. A GPU in the cloud is useful for larger models, peak loads and work for which you temporarily need a lot of capacity. The choice depends on the size of your model, how often you use it and how sensitive your data is.
The trade-offs
| Topic | Own GPU | GPU in the cloud |
|---|---|---|
| Memory | Limited to what the card has | Choice of larger cards and multiple GPUs |
| Usage | Suits regular, predictable use | Suits temporary or varying use |
| Data | Stays on your own machine | Depends on the provider and agreements |
| Management | You manage hardware and software | The provider manages the hardware |
| Flexibility | Fixed hardware | You can switch instances |
Which questions help?
- Does the model fit in the memory of your own card?
- How often and for how long do you use it?
- Is your data allowed to leave the building?
- Do you want to manage it yourself, or would you rather outsource it?
Start small
First test with a small model and a small instance. Measure what you really need before you buy hardware or sign a large contract. Read our overview in renting or buying a GPU and the checklist for choosing a GPU instance.
How to go about it
Start with a small trial: run a small model on your own machine or on a small cloud instance and measure memory, speed and usage. Only scale up once you know what you need. Switch cloud instances off when you are not using them, so you do not pay for idle time.
Also record where your models and data are stored and how you can take them with you. That way you can switch later between your own hardware and a cloud provider without starting over.
Does an AI model fit in your graphics card’s memory? Here is how to work it out
The GPU’s memory is often the first limit. You can make a rough estimate yourself: number of parameters × bytes per parameter. That gives the size of the model weights. The number of parameters and the precision used are usually given in the model’s description.
The Hugging Face documentation on quantization gives the storage per parameter for each precision:
| Precision | Bytes per parameter |
|---|---|
| float32 | 4 |
| float16 or bfloat16 | 2 |
| int8 | 1 |
| int4 | 0.5 |
A model with P billion parameters in 16-bit therefore takes up roughly P × 2 GB of weights. For P, fill in your own model. On top of that comes working memory for input and intermediate results; how much that is differs per program and setting. So allow some spare room. If the model barely fits, a smaller variant or a lower precision may help, or a cloud instance with more memory.
Quantization stores weights with fewer bits, so the model needs less memory. The documentation mentions as the downside that accuracy may decrease, especially at low precision. So test whether the quality is still good enough for your purpose. A lower precision is a trade-off between memory and quality, not a fixed rule. Start with the model in the precision that fits, and only then see whether a smaller variant is needed.
Your own GPU or the cloud: which situation suits which choice?
A few recognisable situations, without there being one right answer:
- You want to learn and experiment. If the model fits in the memory of your own card, that is an easy way to start.
- Your data must not leave the house or office. Then processing locally is the obvious option, or an environment whose terms you know.
- You occasionally have a short, heavy job. An instance that you switch on only for that job can suit better than hardware that sits idle the rest of the time.
- The model only fits when split across several GPUs. Then the connection between those GPUs also plays a part; read GPU interconnects explained.
What do you check before you make a choice?
- Software support. Check whether the program you want to use supports your GPU or the chosen instance. According to Hugging Face, some techniques, such as FP8, need newer hardware.
- Memory. Compare your estimate with the specifications; see GPU specifications explained and the GPU overviews.
- Data. Does your data contain other people’s personal data? Then ask an expert or the Autoriteit Persoonsgegevens (the Dutch Data Protection Authority) for advice on what is allowed and which agreements are needed.
- Portability. Note which versions and settings you use, so that your work can be reproduced on another machine.
Frequently asked questions
Can I test on my own card first?
Yes, that is a good approach as long as the model fits in memory.
What about privacy?
Read our article on AI, data and privacy.
What is quantization in AI models?
With quantization, a model’s weights are stored with fewer bits, for example 8 or 4 instead of 16 or 32. That saves memory, but can reduce accuracy.
How much memory does an AI model need?
As a rough estimate: number of parameters times bytes per parameter, plus extra working memory. The model’s precision determines the bytes per parameter. Check the documentation of the model and of the software.
Further reading
Last updated on
