Overview with affiliate links. We receive a commission if you click through via us. Read how
Compare

Designing a cloud architecture for peak load

An architecture for peaks scales automatically, decouples components and protects itself; testing before the peak is essential.

Door de redactie van ClouwinoGepubliceerd Laatst bijgewerkt

Short answer

An architecture for peak load scales components automatically, absorbs repeated work with caching, decouples heavy tasks through queues and protects itself against overload. Test the limits before the peak arrives and determine in advance where the first bottlenecks are, usually at the database.

Who is it for: Developers, devops engineers and technical buyers who design services for varying or unexpectedly high traffic.

What is a peak and where does it break first?

A peak is a short period with much more traffic or work than normal: a campaign, an announcement, a season or an outage elsewhere that sends traffic to you. The difference from steady growth is that you have little time to react.

The first bottleneck is rarely in the web servers, which you can easily multiply. More often a shared component becomes saturated: the database, an external API, a licence limit or a connection pool. Start by finding that weakest point.

Which design choices help?

Make components stateless, so that you can start more of them without changes. Autoscaling adds capacity based on metrics such as processor usage or queue length; watch the start-up time, because new instances need time.

Caching removes repeated work from the chain: answers in a cache or CDN do not burden the database. A queue decouples accepting work from carrying it out, so that a peak becomes a longer queue instead of an outage. Read replicas and partitioning distribute database load.

How do you protect the system against overload?

Scaling has limits, so protect what cannot grow along. Rate limiting restricts requests per user or client. A circuit breaker stops calls to a failing component, so that a failure does not spread. Graceful degradation lets the service keep running in a slimmed-down form, for example without recommendations, instead of failing completely.

How do you prepare?

Estimate the peak, without pretending you know it exactly. Take a range and build in a margin. Run load tests on an environment that resembles production and see where the chain breaks. Plan the costs too: scaling up is quick, scaling down must also happen automatically, otherwise the bill stays high.

  1. Determine the expected peak as a range and the weakest component.
  2. Test with a realistic load and note where the first errors occur.
  3. Set autoscaling, limits and alerts based on that test.
  4. Practise a failure during the peak and check whether the service keeps running in a slimmed-down form.
  5. Evaluate afterwards and adjust the limits.

Techniques against peaks

What each technique solves and what to watch.
TechniqueWhat does it solvePoint of attention
AutoscalingAdds capacity under loadStart-up time and maximum limits
Caching or CDNReduces repeated workOutdated content and invalidation
QueueAbsorbs work and decouples itProcessing delay and duplicate processing
Rate limitingProtects against overloadWho gets refused and how you see that
Read replicasSpreads read traffic over copiesReplication lag

Who is it not ideal for?

Frequently asked questions

Is reserving capacity in advance better than autoscaling?

For a known peak at a fixed moment, scaling up in advance can be more reliable, because start-up time then plays no role. For unexpected peaks you need autoscaling.

What is a thundering herd?

A situation in which many clients repeat the same action at the same time, for example after a cache has been emptied. Spreading requests over time and locking help prevent that.

How do I test without burdening production?

Use a separate environment with a comparable set-up and synthetic load, or a small, controlled fraction of real traffic.

What do I do about costs during a peak?

Set budget alerts and upper limits, and afterwards look at which scaling step incurred the most costs.

Ask us
Compare Get in touch

Are you sure?