Load balancing and statelessness
What happens when one server is not enough?
Why did the lunch rush log everyone out?
The kitchen now sends out 2,000 boxes a day across Pune. Almost all of them are ordered between 11:30 and 12:30, and almost every customer then keeps the tracking page open.
At 12:31 one day the kitchen screen stops refreshing. Forty boxes are packed and nobody can see which rider takes which. The server’s dashboard shows CPU at 100%, and requests waiting in a queue that grows every second.
The quick fix is to move the app to a bigger machine: copy it over, change one setting and restart it. The site comes back in under a minute.
Then the phone starts ringing. A student halfway through an order has been logged out, and his cart is empty. So is everyone else’s. The app kept each customer’s login and cart in the server’s memory, and a restart wipes memory.
A kitchen that cannot keep up does not buy a bigger stove. It calls a second cook. But a second cook does not know what the first one was told, unless the orders are written on the board instead of kept in one cook’s head.
Before reading on, say what a second server would need to know to answer a customer the first server has been serving. Then say where that information lives today.
Pause and predict The app keeps each customer’s login and cart in one server’s memory. What must move out of that memory before a second server can share the work?
One server has a ceiling
Count the lunch rush first. At 12:30 about 1,800 customers have the app open, and the tracking page asks for an update every five seconds: 1,800 / 5 = 360 requests a second. Menus and new orders add the rest, about 450 requests a second at the peak.
Pause and predict The server can serve about 400 requests a second. At 450 a second, what happens to the wait?
There are two ways to raise the number of requests a second the system can handle.
- Vertical scaling
- Replace the server with a bigger one: more cores, more memory.
- For example: The move to a bigger machine at 12:31. Quick, no code changes, and it needed a restart.
- Horizontal scaling
- Run several copies of the same server and share the requests between them.
- For example: Three servers each taking about 150 of the 450 requests a second.
Vertical scaling is a fine first step; many systems run for years on one large machine. But it does not fix the second failure. Any restart, crash or deploy of the one server still drops every session. That needs horizontal scaling and one more idea: statelessness.
A mental model
- Stateless server
- A server that keeps nothing about a customer between requests. Everything it needs comes in the request or from a shared store.
- For example: A customer’s cart lives in a session store keyed by the session id. Request one goes to server 1, request two to server 3; both read the same cart.
The load balancer itself makes three decisions.
- Who is healthy. A health check asks each server a cheap question (for example
GET /healthz) every few seconds. After a few failures in a row, the server is taken out of rotation; after a few successes it comes back. - Which healthy server gets this request. Round robin takes turns. Least connections picks the server with the fewest requests in flight, which helps when some requests are much slower than others. Hashing on something in the request (the client address, a user id) sends the same client to the same server.
- How deep to look. A layer 4 balancer sees only connections (addresses and ports) and forwards bytes. A layer 7 balancer reads the HTTP request, so it can route by path or header, end TLS, and retry an idempotent request on another server.
How many servers, and how much headroom
The simulator treats each server as a single queue: a server with capacity 400 requests a second that receives λ requests a second spends, on average, 1 / (400 − λ) seconds per request. That one formula explains the curve.
| Servers | Each receives | Busy share | p99 latency |
|---|---|---|---|
| 1 | 450/s | 112% | unbounded: the queue grows |
| 2 | 225/s | 56% | 31 ms |
| 3 | 150/s | 38% | 23 ms |
| 3, one dead | 225/s | 56% | 31 ms |
Two servers carry the rush. So why three? Because of the last row. With three servers, losing one leaves two, and two still carry the peak. With two servers, losing one leaves one at 112%, which is the outage you started with.
The headroom costs money. Three servers at 38% busy look wasteful on a Tuesday afternoon. They are what lets one server die at 12:30 without any customer noticing.
Where should a customer’s cart live?
There is a cheap way to keep sessions in memory with several servers: make the load balancer send each customer to the same server every time. It needs no code changes this week. The cost is that when a server dies, every customer pinned to it loses their cart, even if they were halfway through paying.
Sticky sessions also weaken the balancer. When stickiness is by network address, the office that orders 40 boxes at once sits behind one address, so all 40 people land on one server, which runs hot while the others idle. And the store is not free: it is a new component that must itself be reliable, which Part 3 will deal with properly.
What to remember
- Vertical scaling buys a bigger machine; horizontal scaling adds identical machines behind a load balancer, which also survives the loss of one.
- A load balancer decides who is healthy (health checks), who gets the request (round robin, least connections, hashing), and how deep to look (layer 4 connections or layer 7 HTTP).
- Horizontal scaling only works with stateless servers: sessions and carts live in a shared store, so any server can serve any request and any server can die.
- Size for the peak with one server missing (N + 1), and keep servers well below 100% busy: latency rises steeply as load nears capacity.
Keep state off the app servers, and size for the peak with one server gone.
In the real world
The simulator assumes requests split evenly and each server is a single queue. Real balancers see uneven requests (one slow report among fast page loads), servers with many cores, and health checks that lag behind reality by a few seconds. That is why least-connections and similar algorithms exist, and why real systems often run two load balancers so the balancer itself is not the single point of failure.
The basic design stays the same in real systems: identical servers, state kept outside them, and enough spare capacity to lose one server at the busiest moment.