Logs, streams and back-pressure
What is the difference between a queue and a log, and what happens when consumers fall behind?
Why did the kitchen screen lose half the orders?
The delivery service now has a data team of two. Their first job is a dashboard for the accounts: orders per kitchen, per dish and per hour, updated as orders arrive.
Orders already flow through the queue from lesson 4.1. The order service puts each new order on it, and the kitchen screen takes them off. The data team pointed their analytics job at the same queue, because it was the easiest thing to do.
At 12:20 the next Monday the kitchen screen showed about half the orders. Cooks stood idle while customers rang to ask where lunch was.
Nothing was broken. A queue hands each message to one reader and deletes it once that reader says it is done. With two readers on one queue, each order went to whichever reader asked first. The kitchen got about half of the orders and the dashboard got the other half.
The fix that week gave the analytics job a feed of its own. The order service kept unsent events in a list in its memory and passed them on as fast as the job would take them. It worked until Thursday. The job could process 250 events a second, and the lunch rush produced 400. The list therefore grew by 400 − 250 = 150 events every second, and at 12:31 the order service was killed for using too much memory. About 100,000 events went with it.
A week later the monthly report had a mistake, and the accountant asked to read last Thursday’s orders again. There was no way to. Once a reader had taken an event, it was gone.
Both failures come from needs that a queue was not built for. The kitchen and the dashboard each need every order, in the order they were placed, each at its own speed. And a reader that falls behind must never be able to bring down the service that writes the orders.
Pause and predict The kitchen and the dashboard each need every order, in the order they were placed, at their own pace, and the accountant wants to read last Thursday’s orders again. Which design meets all three needs?
A work queue shares out; a fact must reach everyone
The queue from lesson 4.1 is a work queue. Its job is to share tasks out: each message goes to exactly one of the workers reading it, and it is deleted when that worker acknowledges it. Add a worker and the work is split further. That is exactly right for “send this SMS”, which must be done once, by any one worker.
An order is a task for the kitchen and also a fact that several parts of the business want to know about: the kitchen, the dashboard, the delivery planner next month. If a second reader is pointed at a work queue, the two readers split the orders between them, and neither one gets every order.
The usual first fix is a copy per reader: the producer, or the broker, puts each event on one queue per subscriber. That works, and many systems run this way, but it still has three problems:
- No history. A new reader starts from today. The corrected report cannot read last Thursday again, because every copy was deleted when it was read.
- The writer must know its readers. Every new reader means a new queue wired up on the producing side.
- When a reader is slower than the writer, the extra events have to be stored somewhere. The first fix stored them in the producer’s memory, with no limit. That caused the second failure, and it is the more dangerous of the two.
A mental model
A kitchen already uses both arrangements, on paper.
- Log (append-only log)
- An ordered sequence of records that can only be added to at the end. Readers do not remove anything; each one remembers how far it has read.
- For example: Order 18,442 is appended at offset 9,120,577. The kitchen screen is at offset 9,120,570, the dashboard at 9,050,112. Both will reach it, at their own pace.
In code, a work queue is the FIFO from queues and deques in Coding Patterns: take from the front, and it is gone. A log is a FIFO from which reading never removes anything; each reader keeps its own index into it.
| Work queue | Log | |
|---|---|---|
| Who gets a message | One consumer | Every reader that wants it |
| When it is deleted | When a consumer acknowledges it | When it is older than the retention period, read or not |
| Re-reading old messages | Not possible | Replay: move your position back |
| A new reader | Needs a new queue, starts empty | Starts at the oldest kept offset, or at the end |
| The broker tracks | Every message’s state | One position per reader |
| Good for | Tasks to do once: send this SMS, resize this photo | Facts many parts need: orders placed, payments made |
| Examples | Amazon SQS, RabbitMQ queues | Apache Kafka, Amazon Kinesis Data Streams, Redis Streams |
Kreps names a second property that matters as much as keeping records in sequence: “the log also acts as a buffer that makes data production asynchronous from data consumption.” The writer appends at its own speed, and each reader reads at its own speed. That is what fixes the second failure. To see why, we first need the idea of back-pressure.
The rush in numbers
| Quantity | Working | Result |
|---|---|---|
| Surplus during the rush | 400 − 250 | 150 events/s |
| Memory growth, 1 KB events | 150 x 1 KB | 150 KB/s |
| Time to fill a 100 MB budget | 100,000 / 150 | 667 s ≈ 11 min: 12:31 |
| Backlog at 12:35, rush over | 150 x 900 s | 135,000 events, 135 MB |
| Wait for an event appended at 12:35 | 135,000 / 250 | 540 s = 9 min |
| Spare capacity after the rush, when 100 events/s arrive | 250 − 100 | 150 events/s |
| Time to catch up | 135,000 / 150 | 900 s = 15 min: 12:50 |
The table shows two things. First, the backlog grows at the difference between the rates, so a reader only slightly too slow still fails if the rush lasts long enough. Second, recovery runs on spare capacity: the reader catches up at 250 − 100 = 150 a second, not at 250. The AWS Builders’ Library makes the same point: after an outage, “recovering from the outage requires double the system’s capacity for another hour after the recovery.” Size readers for the catch-up as well as the average.
Now the log’s cost. Suppose the business produces about 5 million events a day at 1 KB each: 5 GB a day, 35 GB for a week of retention. That fits on one ordinary disk. A batch reader replaying one full day at 2,000 events a second takes 5,000,000 / 2,000 = 2,500 s, about 42 minutes. The corrected report can read last Thursday again before lunch.
A queue per reader, or one log?
Two legitimate goals pull against each other here: running as few kinds of system as possible, and giving every reader every event with a week of history behind it.
A queue per reader is a real design, and for work like SMS sends it stays the right one: the kitchen’s tasks keep their work queue. What decides it for orders is replay and new readers. Both were asked for in the first week, and a queue cannot provide either.
What to remember
- A work queue gives each message to one consumer and deletes it on acknowledgement: right for tasks done once. A log keeps messages in order for every reader, each with its own offset: right for facts many parts need.
- A log adds replay and new readers for free: move the position back, or start a new reader at the oldest offset still kept.
- When a consumer is slower than its producer, the gap must be held, refused or lost. Back-pressure decides deliberately which of these happens, and where; an unbounded buffer leads to a crash when memory runs out.
- Block when the producer can wait, drop when old data is worthless, throttle when the producer can be told no, and use a durable log when every event matters and the producer must never wait.
- A backlog grows at the difference of the rates and drains at the spare capacity. Alarm on the age of the oldest unread message.
When the reader is slower than the writer, choose deliberately where the backlog is kept, before it fills the producer’s memory.
In the real world
The simulator has one producer, one reader and smooth rates. Real logs add what the next lesson is about: splitting the log into partitions so many machines can write and read it, copies on several brokers so a dead disk loses nothing, and groups of readers in which each reader takes some of the partitions. Real readers also process in batches, which changes the rates without changing the arithmetic of backlog and catch-up.
Back-pressure appears at every layer: TCP’s receive window, bounded thread pools, the Reactive Streams “request n” protocol, and the crawler’s frontier in lesson 5.6. The question is always the same: when the reader is slower, who waits?