Calculating AI infrastructure costs: GPUs, storage and network
A calculation model for AI costs: GPU hours, storage, network traffic, management and the costs that are often forgotten.
Add up the costs of AI in five groups: compute (GPU hours), storage, network traffic, software and management, and the cost of idle time or trial runs. For each group, calculate usage times the rates you have checked yourself with providers, and allow a margin for unexpected use. Then measure the actual cost per request.
Who is it for: Teams and budget holders who want to estimate the costs of an AI project.
The five cost items
- Compute: number of GPUs times number of hours, for training, fine-tuning and inference.
- Storage: datasets, checkpoints, models and backups, by amount and time.
- Network: outgoing data traffic and traffic between locations.
- Software and management: licences, monitoring, security and your team’s time.
- Trial runs and idle time: failed runs, experiments and machines that are left on.
A calculation model in steps
Start with the workload: how many GPU hours in a month, what amount of data and what traffic. Multiply each by the provider’s rate and add up. Allow a generous margin for trial runs and unexpected use, unless your own measurements justify a different margin. Use our cost calculator to compare options side by side.
Cost per request
For inference, a useful measure is the cost per request or per thousand requests. Divide the monthly costs by the number of requests. This shows you whether an application is economically viable and what smaller models or more efficient hardware would deliver.
Forgotten costs
- Outgoing data traffic when moving large datasets.
- Storage of old checkpoints and copies.
- Machines that keep running after an experiment.
- Minimum commitments or reservations that you do not fully use.
- People’s time for management and maintenance.
How to limit costs
- Choose the smallest model that reaches the quality you need.
- Use temporary or interruptible capacity for work that can resume.
- Switch environments off automatically.
- Measure usage per project and give teams a budget.
- Check prices and terms regularly; they change.
Cost items for AI
| Item | What you measure | Pitfall |
|---|---|---|
| GPU hours | Number of GPUs times hours | Idle machines |
| Storage | Amount of data times time | Old copies and checkpoints |
| Network | Outgoing and between locations | Moving large datasets |
| Management | Your team’s hours and software | Underestimated in plans |
| Trial runs | Failed and experimental runs | Not included in the budget |
Who is this less suitable for?
- Anyone who only uses an AI service with a fixed subscription.
Frequently asked questions
How much margin should I allow?
A generous margin is a common starting point for a budget. Adjust it using your own measurements.
What is the biggest surprise?
Often the sum of small items: storage, traffic and forgotten machines.
How do I compare renting and buying?
Calculate both over the same period and with the same hours of use. See the guide on renting or buying GPUs.
