Hosting AI in production: from prototype to reliable service
What changes when you take an AI model from prototype to production: capacity, monitoring, security and recovery.
A prototype works for one user; production has to be fast, available and secure for many users. That calls for scalable capacity, monitoring of speed and quality, security of input and output, version control of models and a plan for failures and costs.
Who is it for: Teams taking an AI application from experiment to real use.
What changes in production?
A prototype does not need to be fast or reliable and often runs on a single machine. In production, users expect quick answers and availability, and you have to deal with peaks, errors and misuse.
Capacity and scale
Determine how many requests you expect and how many a single GPU or machine can handle. Scale up and down automatically with load, but allow for the start-up time of large models. Plan a minimum of capacity so that the first requests do not have to wait. See the guide on capacity and peak load.
Reliability and recovery
- Run more than one instance and spread them across multiple locations if availability is important.
- Decide what happens if the model or the service goes down: a simple fallback option is better than an error message.
- Store models and configuration with versions, so you can roll back quickly.
- Test recovery just as you would for any other service.
Monitoring and quality
Measure response time, error rate and use of GPU and memory. Also measure the quality of responses: a model can work technically and still give poor results. Collect feedback and check by sampling.
Security and privacy
- Limit who can call the model and use limits against misuse.
- Check what is stored in logs; input may contain personal data.
- Secure models and data with encryption and access rights.
- Allow for attacks that try to mislead the model through the input.
Keeping costs under control
Production runs continuously. Measure the cost per request, switch off unused environments and choose the smallest model size that achieves the quality.
From prototype to production
| Topic | Prototype | Production |
|---|---|---|
| Capacity | One machine | Scalable, multiple instances |
| Availability | No requirement | Agreed and monitored |
| Monitoring | Manual | Measurements and alerts |
| Security | Limited | Access, limits and logging |
| Recovery | Start again | Versions and a recovery plan |
Who is this less suitable for?
- People who only experiment internally without real users.
Frequently asked questions
How many GPUs do I need?
That depends on the model and the number of requests. Measure the throughput of a single GPU with a trial and calculate from there.
Do I have to host it myself?
No. A managed service can be simpler. Choose based on data, control and costs.
How do I keep the quality up?
Measure responses, collect feedback and test after every change to the model.
