Overview with affiliate links. We receive a commission if you click through via us. Read how
Compare

Cloud storage for backups, archives and data lakes

Backups, archives and data lakes use the same storage technology but call for a different set-up and different checkpoints.

Door de redactie van ClouwinoGepubliceerd Laatst bijgewerkt

Short answer

Backups, archives and data lakes are three different purposes for cloud storage. Backups must be recoverable and protected, archives kept for a long time at low cost and data lakes quickly searchable. For each purpose, choose the access pattern, the retention period and the security measures.

Who is it for: DevOps teams, data engineers and technical buyers who set up cloud storage for retaining and analysing data.

How do the three purposes differ?

A backup is a copy you need when the original is lost or damaged. An archive keeps data for a long time because it has to be kept or is valuable, but you rarely read it. A data lake is a large collection of raw and processed data on which you run analyses.

All three often fit on object storage, because it scales well and is charged per amount of data. The difference lies in the set-up: how often do you read, how long do you keep and who may change or delete?

What does a backup require of the storage?

The most important thing is recoverability. A backup that the same administrator or account can erase, just like the original, offers poor protection against ransomware or a faulty command. Immutable storage (object lock) and a separate management environment reduce that risk.

Also think about restore speed. Restoring large datasets requires throughput and can incur outbound traffic costs. Practise a full restore and measure how long it takes, because that time determines what you call an acceptable interruption.

What does an archive or a data lake require?

An archive is about retention period and costs. Cold storage classes lower the price of keeping data, but raise the price or delay of retrieval. Use lifecycle rules to record when data moves and when it may disappear. Check how you demonstrate the integrity of old data, for example with checksums.

A data lake requires a good structure: logical prefixes, partitioning by date or source and file formats that analysis tools read efficiently. Many small files slow down processing and increase the number of requests. Add a catalogue that describes what is stored and who has access.

How do you combine them without stacking costs or risk?

Use separate buckets or projects per purpose, with their own permissions and retention. Do not let a backup serve as an archive or data lake at the same time, because the requirements clash: a lake must be changeable, a backup precisely not.

Three purposes side by side

Typical requirements; work them out for your own situation.
PurposeAccess patternKey measure
BackupRarely read, sometimes fast and complete restoreImmutability, separate management, restore test
ArchiveVery rarely read, long retention periodCold class, lifecycle, integrity check
Data lakeFrequent and parallel reads by analysis toolsStructure, catalogue, access management
RegulationDepends on sector and dataRecord retention period and location in advance

Who is it not ideal for?

Frequently asked questions

Is a snapshot a backup?

A snapshot is usually in the same environment as the original. Only with a separate copy in another location do you call it a backup.

Which file format suits a data lake?

Column-oriented formats let analysis tools read only the columns they need. For raw input, a simple text format is often more practical.

How long should I keep archive data?

That depends on laws and regulations and on your sector. Have a lawyer or compliance officer determine this and record it in lifecycle rules.

What is a restore test?

It is a planned exercise in which you actually restore data into a test environment. That way you discover flaws before an incident occurs.

Ask us
Compare Get in touch

Are you sure?