Design a message queue

Why did the kitchen stop for one slow van?

It is festival day, three weeks after the broker failure. The delivery service now runs as a handful of services: orders, the kitchen screens, packing, rider dispatch and invoicing. They still talk the way they were first wired. When an order is placed, the order service calls the kitchen service. The kitchen service calls packing before it answers. Packing calls dispatch to book a rider before it answers.

At 12:34 the rider cooperative’s system slows down, and every dispatch call takes eight seconds. Packing’s calls time out and retry. The kitchen service waits on packing. The order service waits on the kitchen. At 12:36 customers watch the Order button spin.

Late vans are normal on a festival day. A kitchen that stops taking orders because of them should not be. When a van is late, the cooks keep cooking and the boxes wait at the counter. Here a slow rider booking has reached all the way back to the Order button.

The order log from lesson 5.2 already exists. Today it feeds only the packing screen and the data team. The other design is to have every step write what happened to the log and let every team read it at its own pace, so that nothing calls anything.

That design must answer three questions before it can replace the chain. The cooks must see tickets in the order they came, and a cancellation must never reach them before its order. One broken ticket must not stop the whole line. And when a famous cook posts a special and her kitchen takes a quarter of all orders, her tickets must not queue behind one overloaded consumer, the program that reads the log for a team. Make a guess before reading on: what must an order and its cancellation share for the log to keep them in sequence?

Pause and predict Every step now writes to the log. A cancellation must never reach the cooks before its order. What must the two events share?

What must never go wrong?

The obvious version, and how it breaks

The first version is a chain of HTTP calls, each service waiting for the next before it answers. It is easy to read and easy to debug: one request, one stack trace.

Pause and predict Four services, each up 99.9% of the time, called in a chain. How often are all four up together?

It works on quiet days. On festival day it breaks in three ways, all of which you can compute:

• Latency adds up. The order cannot be confirmed until a rider is booked. With dispatch at 8 s, every order waits at least 8 s, even though the customer does not care which rider takes the box.
• Availability multiplies. Four services that are each up 99.9% of the time, called in a chain, are all up together 0.999⁴ = 99.6% of the time: four times the downtime of any one.
• Retries pile up. Packing retries dispatch, the kitchen retries packing, the order service retries the kitchen. One slow service at the bottom turns into many times its load from above (lesson 4.1).

None of these steps needs an answer from the next one. The order service needs to know the order is safely recorded; the kitchen needs to hear about it soon. A log fits that need better than a call does: the order service writes the order once, and each team reads it when it is ready.

A mental model

A parcel station ran something like this long before software did.

Key by kitchen or by order?

Two goals compete: the simplest possible kitchen screen, and a key that survives the busiest kitchen on the busiest day.

The kitchen’s requirement is real, and on a day without a famous cook the kitchen key is the simpler design. But the key must survive the busiest kitchen on the busiest day, and a kitchen key ties one kitchen’s whole festival to one consumer. The kitchen can still see its tickets in order, with the order coming from somewhere else: the order service already writes the order in a transaction, so it can number each kitchen’s tickets there, and the screen shows them in that order, holding a slot open for a second if a ticket is late.

What to remember

1. Replace a chain of waiting calls with facts on a log: producers publish what happened and return; every team reads in its own consumer group at its own pace.
2. The key decides both the spread and the ordering. Key by the entity whose events must stay in sequence, and check that no single key can carry more than one consumer can process.
3. Size partitions from the slowest consumer: events per second times time per event, with headroom. Partitions bound parallelism, cannot be reduced, and adding them reshuffles keys.
4. A poison event must not block its partition: a few quick retries, then retry topics per team, then a dead-letter topic with an alert and a re-drive. Pause instead when the dependency, not the event, is broken.
5. Delivery is at least once, so every consumer is idempotent: record the event id with the work, in one local transaction.

Publish what happened and return; a slow team should build its own lag, never anyone else’s.

In the real world

Real pipelines add what our toy leaves out: a schema registry so that producers cannot publish an event its readers cannot parse, quotas so that one producer cannot flood the cluster, consumer lag dashboards per group in seconds rather than events, and tooling to inspect and re-drive dead letters. Teams that need retries with delays often use a library or framework that builds the retry-topic chain for them.

The order service still has one hard problem: it writes the order to its database and publishes OrderPlaced to the log, two systems, and a crash between the two loses one of them. That is the dual-write problem, and the outbox in lesson 6.5 solves it.