Overview with affiliate links. We receive a commission if you click through via us. Read how
Compare

Running AI: At Home on Your Own GPU or in the Cloud?

The trade-offs between running AI on your own GPU and renting GPU capacity in the cloud.

Blog5 min read

Short answer

Your own graphics card is suitable for experimenting with smaller models and for keeping data in your own hands. A GPU in the cloud is useful for larger models, peak loads and work for which you temporarily need a lot of capacity. The choice depends on the size of your model, how often you use it and how sensitive your data is.

The trade-offs

Own GPU and cloud GPU at a glance
TopicOwn GPUGPU in the cloud
MemoryLimited to what the card hasChoice of larger cards and multiple GPUs
UsageSuits regular, predictable useSuits temporary or varying use
DataStays on your own machineDepends on the provider and agreements
ManagementYou manage hardware and softwareThe provider manages the hardware
FlexibilityFixed hardwareYou can switch instances

Which questions help?

Start small

First test with a small model and a small instance. Measure what you really need before you buy hardware or sign a large contract. Read our overview in renting or buying a GPU and the checklist for choosing a GPU instance.

How to go about it

Start with a small trial: run a small model on your own machine or on a small cloud instance and measure memory, speed and usage. Only scale up once you know what you need. Switch cloud instances off when you are not using them, so you do not pay for idle time.

Also record where your models and data are stored and how you can take them with you. That way you can switch later between your own hardware and a cloud provider without starting over.

Does an AI model fit in your graphics card’s memory? Here is how to work it out

The GPU’s memory is often the first limit. You can make a rough estimate yourself: number of parameters × bytes per parameter. That gives the size of the model weights. The number of parameters and the precision used are usually given in the model’s description.

The Hugging Face documentation on quantization gives the storage per parameter for each precision:

PrecisionBytes per parameter
float324
float16 or bfloat162
int81
int40.5

A model with P billion parameters in 16-bit therefore takes up roughly P × 2 GB of weights. For P, fill in your own model. On top of that comes working memory for input and intermediate results; how much that is differs per program and setting. So allow some spare room. If the model barely fits, a smaller variant or a lower precision may help, or a cloud instance with more memory.

Quantization stores weights with fewer bits, so the model needs less memory. The documentation mentions as the downside that accuracy may decrease, especially at low precision. So test whether the quality is still good enough for your purpose. A lower precision is a trade-off between memory and quality, not a fixed rule. Start with the model in the precision that fits, and only then see whether a smaller variant is needed.

Your own GPU or the cloud: which situation suits which choice?

A few recognisable situations, without there being one right answer:

What do you check before you make a choice?

  1. Software support. Check whether the program you want to use supports your GPU or the chosen instance. According to Hugging Face, some techniques, such as FP8, need newer hardware.
  2. Memory. Compare your estimate with the specifications; see GPU specifications explained and the GPU overviews.
  3. Data. Does your data contain other people’s personal data? Then ask an expert or the Autoriteit Persoonsgegevens (the Dutch Data Protection Authority) for advice on what is allowed and which agreements are needed.
  4. Portability. Note which versions and settings you use, so that your work can be reproduced on another machine.

Frequently asked questions

Can I test on my own card first?

Yes, that is a good approach as long as the model fits in memory.

What about privacy?

Read our article on AI, data and privacy.

What is quantization in AI models?

With quantization, a model’s weights are stored with fewer bits, for example 8 or 4 instead of 16 or 32. That saves memory, but can reduce accuracy.

How much memory does an AI model need?

As a rough estimate: number of parameters times bytes per parameter, plus extra working memory. The model’s precision determines the bytes per parameter. Check the documentation of the model and of the software.

Further reading

Last updated on

Ask us
Compare Get in touch

Are you sure?