Real-time delivery

Why are 40,000 phones asking the same question every five seconds?

By winter the delivery service has 2,000 home cooks and 200,000 customers. At lunch most of them do the same thing after ordering: they open the tracking page and watch.

The first tracking page showed the status when it loaded, and nothing after. Customers refreshed it by hand. The ones who had to eat on time, among them a diabetic customer whose insulin is timed to lunch, phoned the kitchen instead.

The fix took a few lines of JavaScript: ask the server for the order’s status every five seconds. The phone calls stopped.

Two weeks later the lunch-hour graph shows what this costs. Forty thousand phones have the page open at 12:45, and the servers answer eight thousand requests a second. A box changes status about once every half minute.

Do the sum before reading on. A phone that asks every five seconds about an order that changes every thirty seconds gets the answer it already had five times in six. The logs agree: five answers in six say nothing has changed. Eight thousand questions a second deliver about thirteen hundred pieces of news.

Polling every fifteen seconds instead would cut the load to a third. It would also show “rider at your door” fifteen seconds after the rider knocks, and the phone calls would come back.

On the web the phone normally speaks first: it sends a request and the server answers. The server is the side that knows when the status changes, but it cannot tell the phone until the phone asks. How can the server send the news first, as soon as it has some?

Pause and predict 40,000 phones ask 8,000 times a second to learn about 1,300 changes. What removes the empty answers without making news late?

Polling: a loop of questions

The tracking page uses short polling: every phone sends an ordinary HTTP request on a timer, and the server answers at once with whatever it has. It is the easiest real-time feature to build. Every server, proxy and load balancer already understands it, and the servers stay stateless.

Pause and predict An order’s status changes on average every 30 seconds, at random. The phone asks every 5 seconds. What share of the answers say nothing has changed?

Its cost has a simple shape. With N phones each asking every P seconds:

• Requests per second are N / P, whether or not anything happened.
• The delay before a phone sees a change is about P / 2 on average: the change waits for the next question.
• Empty answers are most of them when changes are rarer than questions: about 85% at the tracking page’s settings.

So the poll interval is a single setting that trades load against delay. Halving the delay doubles the requests, and no interval gives both low load and low delay.

There is a cost on the phone too. Each request can wake the phone’s mobile radio, and a radio that keeps waking up drains the battery; Ilya Grigorik’s book on browser networking recommends eliminating periodic small transfers on mobile for exactly this reason.

A mental model

Long polling

The phone sends a request; the server does not answer until it has news or a timeout passes. On an answer, the phone sends the next request at once.

For example: The phone asks “anything after event 40?”. The server waits 12 seconds, the rider picks up the box, and the server answers with event 41. The phone asks again.

Server-sent events (SSE)

A normal HTTP request whose response never ends. The server writes small text events into it whenever it has news; the browser’s EventSource hands each one to the page.

For example: id: 41 then data: picked up, then a blank line. That is one event, a few dozen bytes, with no new request.

WebSocket

A connection that starts as an HTTP request and is then upgraded into a two-way channel: either side can send a message at any time.

For example: A chat window: the cook types a reply while the customer’s next question is already on its way.

Step through one tracking stream, then switch to the failure path to see what a dropped connection costs.

The tracking page in numbers

Forty thousand phones, a change every 30 seconds per order on average (1,333 changes a second in all), an 80 ms round trip, and 700 bytes of HTTP headers per request and response:

Where the long-poll figure comes from: a held request ends at the first change or at 30 seconds, whichever comes first. With changes arriving at random every 30 seconds on average, that takes 30 x (1 − e⁻¹) ≈ 19 seconds, so each phone asks about once every 19 seconds: 40,000 / 19 ≈ 2,100 requests a second.

The headers tell the same story in bytes. Polling every 5 seconds spends 8,000 x 700 B = 5.6 MB a second on headers alone. A stream spends about 16 bytes of framing per event, a 3-byte comment line per phone every 15 seconds, and a full set of headers only when a phone reconnects: about 120 KB a second for the same news.

The other half: who holds the connections

The phone’s stream lands on one machine. The news comes from somewhere else: the rider’s app tells the order service, which may run on any of a dozen machines. So push needs three new pieces.

• A connection gateway: a fleet of small servers whose only job is to hold streams. Keeping business logic out of them means they rarely need redeploying, and every redeploy closes connections.
• Publish/subscribe: the order service publishes “order 8812 picked up” to a channel; every gateway that holds a phone watching order 8812 has subscribed to it. Neither side needs to know where the other is.
• A history of events per order, so a phone that reconnects with Last-Event-ID: 41 can be sent what it missed. Pub/sub of this kind forgets a message once it is delivered; the history has to live in a store.

Load balancers need care too. They balance new connections, so a gateway added at 12:30 gets only the phones that connect after 12:30. Their idle timeouts must be longer than the heartbeat interval, or they will cut quiet streams.

Make the questions cheap, or stop asking?

Two goals compete here. One is to keep every server stateless and the system as small as it is today. The other is to put the news on the phone the moment it exists.

Cheap polling is a sound design, and for a page whose news changes once an hour it would be the right one. Here the news that matters most (“rider at your door”) is worth seconds, the customers who need it are the ones who already phone the kitchen, and Part 4 will need held connections for chat anyway. What makes it affordable is that a stream is opened only while a box is on the road, for a few minutes each, and never for every page view.

What to remember

1. Short polling costs N / P requests a second whether or not anything happened, and delays news by about P / 2. The interval is the only setting, and it trades load against delay.
2. Long polling holds each request until news or a timeout; SSE streams events one way on an open response; WebSocket is a two-way channel after an HTTP upgrade.
3. Push turns requests into held connections: state on a gateway, balanced by connection, drained carefully on deploy, kept alive with heartbeats.
4. A server that pushes must be able to resume: keep a history of events and answer “what happened after event 41?”.
5. Use SSE when the server does the talking, WebSockets when both sides do, and plain polling when news is rare or a few seconds do not matter.

Push the news from the side that has it, and hold a connection open only while there is news to send.

In the real world

The simulator assumes a fixed round trip, no lost packets and a constant header size. Real systems differ in ways that matter: HTTP/2 compresses headers and carries many streams on one connection, mobile networks drop connections far more often at some times than others, and a gateway’s real limit is usually memory per connection and the operating system’s file descriptors, which you measure instead of guessing.

Native apps have one more channel: the phone platform’s own push notification service, which can wake an app that is not running. That is a different job (reaching a phone that is not listening), and it is part of the notification system in lesson 4.5. The tracking stream, the gateways and pub/sub built here carry the chat in lesson 4.4.