Resilience

Why could nobody order when only the maps were slow?

Every order page shows a small map: the kitchen, the customer’s building, and an estimated time. The map and the time come from an outside maps provider, which normally answers in about 80 milliseconds.

At 12:35 one weekday the provider has a bad day. Its answers start taking three seconds.

By 12:37 nobody in the city can place an order. The map is the smallest part of the problem: the menu, the cart and the Pay button all spin, then fail. The order count sits at zero for twenty-three minutes of the busiest hour.

The maps company was slow. The kitchen’s own servers were not. So why did the menu stop?

The code explains it, in two places. The call to maps has no timeout, so a server thread waits as long as maps takes. And when a call does fail, the page retries it three times, and inside each try the maps client library retries three more times on its own.

The usual first reaction is to add retries with backoff and more servers, on the theory that this is a capacity problem. Before reading on, work out the numbers. Each server has 600 request threads. Four hundred pages arrive a second, and each now holds a thread for about three seconds while it waits for maps. How many threads would that need, against the 600 there are? Keep your number; the next section checks it. Whatever it is, doubling the servers would only mean that twice as many threads fill up waiting for maps, a little later.

No kitchen sends every cook to stand at the gate when the vegetable supplier is late, yet that is what the servers did. Every request thread went to wait for maps, and the pages that never needed a map found nobody left to serve them. A slow part had become a total outage.

Pause and predict Maps took three seconds and the whole menu stopped. What is the first fix?

Slow is worse than down

If maps had been switched off, each call would have failed in a millisecond and the page would have shown an error in the map square. That is annoying, but the rest of the page would have kept working. Slow is worse, because a slow call holds something while it waits: a request thread, a database connection, a slot in a pool.

Step through one page. Count where the time goes.

The arithmetic is Little’s law from lesson 4.1: threads in use = pages per second x seconds each page holds a thread.

Pause and predict 400 order pages arrive a second. Each holds a thread for 60 ms of our own work plus the maps call, and there are 600 threads. Maps slows from 80 ms to 3 s. How many threads would the pages need?

Once the pool is full, new pages queue for a thread. The 600 threads can finish only about 600 / 3.06 ≈ 196 pages a second, so the queue grows by about 200 pages a second, and every page waits, including the ones whose own work is 60 ms. That is a cascading failure: the failure grows by feeding on itself.

Cascading failure

A failure that grows over time because of positive feedback: one slow or broken part makes the parts around it slow or broken too, and so on outwards.

For example: Maps slows, threads wait for maps, the shared pool runs out, the menu and checkout queue behind them, phones retry, and the queue grows faster.

The retries made it worse. Each level makes up to 4 attempts (the first try and three retries), and the client library’s attempts happen inside each of the page’s, so a page can call maps up to 4 x 4 = 16 times: a provider already struggling at 400 calls a second is asked for 6,400.

A mental model

The lunch hour in numbers

All figures come from the cascade simulator (“Keep lunch running while maps is slow”): 400 pages a second, 600 threads, 60 ms of our own work per page, maps at 3 s unless stated.

Three things in this table matter. First, the breaking point: with 600 threads, the pool copes while 400 x (0.06 + t) ≤ 600, that is while maps takes under 1.44 s. Below that latency nobody notices anything; a little above it, everything fails. Overload does not get worse gradually; a small change in latency takes the system from fine to failing.

Second, a timeout alone frees the threads but fails every page, because the code treated the map as required. Surviving the slowdown needs both the mechanism (the timeout) and a product decision about which parts of a page are optional.

Third, load shedding cannot fix a slow dependency. It lets about half the pages through and refuses the other half quickly. It protects the server from taking on more work than it can finish, but each page it accepts still waits 3 s for maps.

Goodput

The part of throughput that is useful: requests answered correctly and soon enough for the caller to use the answer.

For example: With no defences the servers are 100% busy, and 89% of the pages they finish arrive after the phone has given up: high throughput, goodput near 11%.

Defend the call, or remove it from the path?

Two goals compete. One is to keep the map on the order page, exactly as customers see it today, and wrap the call in every defence. The other is to make sure the path that saves the page runs every day, because a fallback that runs only during an outage tends to fail on the one day it is needed.

The defended call works, and many teams run exactly that. The second path is the argument the AWS Builders’ Library makes about fallback: a path that runs only during an outage hides bugs until the outage. Making the degraded state the normal state means it is exercised all day. The defences are still needed, on the map endpoint, where a failure can cost only a map.

What to remember

1. A slow dependency is worse than one that is down: a slow call holds a thread, and by Little’s law, longer waits at the same request rate fill the pool. Past that point everything fails at once.
2. Give every network call a timeout set from the dependency’s healthy p99.9, and propagate deadlines.
3. Retry at one layer with a retry budget; a circuit breaker stops calls to a failing dependency, but it only sees failures, so it needs the timeout.
4. A bulkhead gives a risky dependency its own small pool; load shedding refuses excess fast. Both keep capacity free for the rest of the work.
5. Decide what each page can live without, and prefer removing a dependency from the critical path over a fallback that runs only during outages.

Give every outside call a deadline, and never let an optional dependency hold a thread the required work needs.

In the real world

The simulator is a fluid model: every maps call takes exactly the same time, and queues grow without limit. Real latency varies, so a real timeout cuts off only the slow tail; real servers also run out of memory, fail health checks and get restarted by the load balancer, which moves the load to the survivors and can make the cascade faster. Production breakers count failures in sliding windows and often look at latency too; production load shedding usually looks at request age or queue time instead of a fixed capacity.

The pattern returns in the next lessons: the LLM support bot (lesson 7.3) needs per-tenant quotas and shedding when the GPUs are full, and going multi-region (lesson 7.4) applies the same idea, limiting how much one failure can take down, to whole regions.