Data storage for AI: object storage, data lakes and datasets
Which storage suits AI: object storage for datasets, fast storage for training and a good structure for your data.
AI projects usually use object storage for large datasets and archives, and faster storage close to the compute for active work. A data lake keeps raw data in one place. A good structure with version control and access permissions prevents chaos and makes results repeatable.
Who is it for: Teams who want to store and organise data for AI and are looking for the right type of storage.
Which data do you keep?
An AI project works with raw data, processed datasets, intermediate results (checkpoints) and final models. Each has a different use: raw data you keep for a long time, checkpoints often temporarily, models and datasets with versions.
Object storage for large datasets
Object storage is suitable for large numbers of files that you read or write as a whole. It scales well and is usually more cost-effective per amount stored. It is less suitable for many small, rapidly changing files. Read also the guide on object, block and file storage.
Fast storage for active work
During training, data has to flow quickly to the GPUs, otherwise expensive processors end up waiting. For that you use fast local or shared storage, often filled from object storage. Measure whether your storage is the bottleneck before you buy more compute.
Data lakes
A data lake keeps raw data in its original form in one central place, usually on top of object storage. You build datasets from it. The advantage is flexibility. Without agreements on structure, quality and access, it becomes a swamp in which nobody can find anything any more.
Keep it manageable
- Use clear folders and names and record where data comes from.
- Keep versions of datasets, so that you can repeat results.
- Limit access to those who need it and encrypt sensitive data.
- Set retention periods and clear up temporary files.
- Watch outgoing data traffic when you move data between locations or providers.
Which storage for what?
| Storage type | Suitable for | Watch out for |
|---|---|---|
| Object storage | Datasets, archives, models | Less suitable for many small, rapidly changing files |
| Fast shared or local storage | Active training and processing | More expensive per amount stored, often temporary |
| Data lake | Raw data from multiple sources | Requires agreements on structure and access |
| Archive | Old data and results | Retrieval takes time or money |
Who is it not ideal for?
- Anyone who only uses an AI service via an API and does not manage their own dataset.
Frequently asked questions
Do I need a data lake?
Only if you want to bring together data from many sources. For a small project, a tidy folder in object storage is enough.
How do I avoid high costs?
Move data as little as possible, use cheaper storage for old data and clear up temporary files.
Is backup needed for AI data?
Yes, for data and models that are hard to recreate. Keep a copy in another place.
