Designing a cloud architecture for peak load
An architecture for peaks scales automatically, decouples components and protects itself; testing before the peak is essential.
An architecture for peak load scales components automatically, absorbs repeated work with caching, decouples heavy tasks through queues and protects itself against overload. Test the limits before the peak arrives and determine in advance where the first bottlenecks are, usually at the database.
Who is it for: Developers, devops engineers and technical buyers who design services for varying or unexpectedly high traffic.
What is a peak and where does it break first?
A peak is a short period with much more traffic or work than normal: a campaign, an announcement, a season or an outage elsewhere that sends traffic to you. The difference from steady growth is that you have little time to react.
The first bottleneck is rarely in the web servers, which you can easily multiply. More often a shared component becomes saturated: the database, an external API, a licence limit or a connection pool. Start by finding that weakest point.
Which design choices help?
Make components stateless, so that you can start more of them without changes. Autoscaling adds capacity based on metrics such as processor usage or queue length; watch the start-up time, because new instances need time.
Caching removes repeated work from the chain: answers in a cache or CDN do not burden the database. A queue decouples accepting work from carrying it out, so that a peak becomes a longer queue instead of an outage. Read replicas and partitioning distribute database load.
How do you protect the system against overload?
Scaling has limits, so protect what cannot grow along. Rate limiting restricts requests per user or client. A circuit breaker stops calls to a failing component, so that a failure does not spread. Graceful degradation lets the service keep running in a slimmed-down form, for example without recommendations, instead of failing completely.
How do you prepare?
Estimate the peak, without pretending you know it exactly. Take a range and build in a margin. Run load tests on an environment that resembles production and see where the chain breaks. Plan the costs too: scaling up is quick, scaling down must also happen automatically, otherwise the bill stays high.
- Determine the expected peak as a range and the weakest component.
- Test with a realistic load and note where the first errors occur.
- Set autoscaling, limits and alerts based on that test.
- Practise a failure during the peak and check whether the service keeps running in a slimmed-down form.
- Evaluate afterwards and adjust the limits.
Techniques against peaks
| Technique | What does it solve | Point of attention |
|---|---|---|
| Autoscaling | Adds capacity under load | Start-up time and maximum limits |
| Caching or CDN | Reduces repeated work | Outdated content and invalidation |
| Queue | Absorbs work and decouples it | Processing delay and duplicate processing |
| Rate limiting | Protects against overload | Who gets refused and how you see that |
| Read replicas | Spreads read traffic over copies | Replication lag |
Who is it not ideal for?
- Extensive autoscaling is less suited to applications with a stable, predictable load; fixed capacity is then simpler.
- Queues do not suit actions where the user expects a final answer straight away.
- Extra layers such as caches and replicas add complexity that a small team must be able to manage.
Frequently asked questions
Is reserving capacity in advance better than autoscaling?
For a known peak at a fixed moment, scaling up in advance can be more reliable, because start-up time then plays no role. For unexpected peaks you need autoscaling.
What is a thundering herd?
A situation in which many clients repeat the same action at the same time, for example after a cache has been emptied. Spreading requests over time and locking help prevent that.
How do I test without burdening production?
Use a separate environment with a comparable set-up and synthetic load, or a small, controlled fraction of real traffic.
What do I do about costs during a peak?
Set budget alerts and upper limits, and afterwards look at which scaling step incurred the most costs.
