Overview with affiliate links. We receive a commission if you click through via us. Read how
Compare

Data storage for AI: object storage, data lakes and datasets

Which storage suits AI: object storage for datasets, fast storage for training and a good structure for your data.

Door de redactie van ClouwinoGepubliceerd Laatst bijgewerkt

Short answer

AI projects usually use object storage for large datasets and archives, and faster storage close to the compute for active work. A data lake keeps raw data in one place. A good structure with version control and access permissions prevents chaos and makes results repeatable.

Who is it for: Teams who want to store and organise data for AI and are looking for the right type of storage.

Which data do you keep?

An AI project works with raw data, processed datasets, intermediate results (checkpoints) and final models. Each has a different use: raw data you keep for a long time, checkpoints often temporarily, models and datasets with versions.

Object storage for large datasets

Object storage is suitable for large numbers of files that you read or write as a whole. It scales well and is usually more cost-effective per amount stored. It is less suitable for many small, rapidly changing files. Read also the guide on object, block and file storage.

Fast storage for active work

During training, data has to flow quickly to the GPUs, otherwise expensive processors end up waiting. For that you use fast local or shared storage, often filled from object storage. Measure whether your storage is the bottleneck before you buy more compute.

Data lakes

A data lake keeps raw data in its original form in one central place, usually on top of object storage. You build datasets from it. The advantage is flexibility. Without agreements on structure, quality and access, it becomes a swamp in which nobody can find anything any more.

Keep it manageable

Which storage for what?

General guideline.
Storage typeSuitable forWatch out for
Object storageDatasets, archives, modelsLess suitable for many small, rapidly changing files
Fast shared or local storageActive training and processingMore expensive per amount stored, often temporary
Data lakeRaw data from multiple sourcesRequires agreements on structure and access
ArchiveOld data and resultsRetrieval takes time or money

Who is it not ideal for?

Frequently asked questions

Do I need a data lake?

Only if you want to bring together data from many sources. For a small project, a tidy folder in object storage is enough.

How do I avoid high costs?

Move data as little as possible, use cheaper storage for old data and clear up temporary files.

Is backup needed for AI data?

Yes, for data and models that are hard to recreate. Keep a copy in another place.

Ask us
Compare Get in touch

Are you sure?