Delta infrastructure · Learn & Share

Why new capacity takes five minutes, and why that is the design

When traffic spikes, a new web instance is serving users about five minutes later. That number surprises people, so this is a short tour of where it goes, why chasing it is the wrong instinct, and what genuinely large services do instead.

Where the five minutes goes

Two components, and they are not the ones people guess. Detection is ~3.5 minutes: CloudWatch needs three consecutive one-minute readings above the scaling target, plus up to a minute of metric lag. That window is an AWS constant for target tracking, and it exists for a reason: it is what separates "sustained demand" from "one hot minute", so we do not pay for instances a bot scan asked for. Boot is ~1.5 minutes, measured: our instances go from launch call to serving real traffic in 90–105 seconds.

Notice which term dominates. Halving boot time, real engineering work, would improve the total by about 15%. This is why "make instances start faster" is rarely where mature teams spend effort.

So why not detect faster? We could, by about two minutes: the three-reading window is fixed inside AWS's managed target-tracking alarms, so going faster means replacing them with hand-tuned alarms that fire on a single hot minute. We have chosen not to, because one minute of data cannot tell a real spike from a blip: a crawler burst, one heavy report, a cache refresh after a deploy all look identical to the first minute of genuine load. Three minutes is the price of scaling on demand instead of noise, and the headroom numbers below are why those two minutes cost us nothing in practice.

What actually happens to users during those five minutes

Usually: nothing, because of the gap between normal load and standing capacity. During school hours we run a minimum of two instances, each comfortable at 15–20 req/s, while normal traffic is around 1–3 req/s in total; our highest observed peak this term is 6.9 req/s on one instance. That headroom means a spike must reach many times normal traffic before users feel anything at all, and the scale-out trigger sits well inside it (5 req/s per instance, set deliberately early for now while we validate the new request-count policy against real traffic), so new capacity is normally on its way long before the existing capacity is under strain.

If one ever does, degradation is gradual, not a cliff. Full instances queue requests rather than reject them: pages get slower before anything errors, alarms reach Slack within minutes, new capacity lands, and the system heals itself. The realistic worst case for an unforeseeable, enormous spike is a few minutes of slow pages and some 502s for part of the wave, not an outage.

The hierarchy big services actually use

The instinct "five minutes is too slow, detect faster" misreads how scaling works at every size. Detection latency is nearly universal (Kubernetes and ECS inherit the same one-to-three-minute metric loop; containers shrink boot, not detection). So the services with the scariest traffic do not detect faster. They arrange things so detection speed stops mattering:

1. Standing headroom we do this
   Run enough capacity that surges are absorbed by machines already on. Netflix-scale fleets sit at 40–60% utilisation on purpose. Our school-hours minimum of two is the same idea at our size.
2. Prediction we do this, by schedule
   Scale on what you know is coming, not on what already arrived. The giants learn their forecast from traffic models (true predictive autoscaling, which we have just switched on in a forecast-only trial; the note below explains); ours is currently hand-encoded: a scheduled weekday capacity window, plus event-scale.sh to pre-warm capacity before a known event (exam results, a marketing push) so the wave lands on instances already serving.
3. Load shedding & graceful degradation
   Accept that scaling sometimes loses the race, and build the application to degrade instead of queue: serve the video without the recommendations panel, return cached data, fast-fail the overflow with a retry. This is the biggest cultural difference at Netflix scale, and it is application architecture, not infrastructure.
4. Per-request scheduling
   Serverless platforms (Lambda and friends) have no detection loop at all: the request itself is the signal, and a micro-VM is allocated in ~150 ms. The trade is cold-start tails and pricing that inverts badly for steady traffic like ours.

Reactive autoscaling, the thing the five minutes belongs to, is the safety net under all four layers, at every scale. It is nobody's first line of defence.

How far are we from "Netflix level"?

Structurally, closer than the size difference suggests: we already run layers 1 and 2, sized to our traffic, with the reactive net underneath. What Netflix adds is layers 3 and 4 plus enormous standing fleets, and they add them because their volume makes those costs pay. For us the same layers would be complexity collected daily for benefits collected never. The gap is a deliberate choice, not a backlog. If our traffic ever grew a hundredfold, the first genuinely new thing worth building would be layer 3, graceful degradation, because that is what turns "the scaling lost the race" from an incident into a slightly worse page.