Inside Kafka
How does Kafka keep millions of messages in order, survive a dead broker, and avoid processing one twice?
Why did the packing screen show order 8812 twice?
Since the last lesson, every order goes into one log. It runs on Apache Kafka: three brokers (the servers that store the log), and a topic called orders split into six partitions. The kitchen screen, the packing screen and the data team each read it at their own pace.
At 12:40 on a Thursday, the busiest ten minutes of the day, broker 2’s disk controller fails. Twenty seconds later the packing screen shows order 8812 twice, and about a dozen other orders the same way. Two packers are filling the same tiffin.
The order database says each of those orders was placed once and paid once. The duplicates are in the log itself: the same order, at two different offsets.
The order service, which is the producer that writes orders to the log, is set to acks=all. With that setting, each write waits until every in-sync copy of the partition has stored the order. It was chosen because it was supposed to be the safe setting, yet a dozen orders were stored twice. Until we know what acks=all actually guarantees, we cannot be sure that no order was lost either.
Kafka also offers exactly-once delivery as a setting. Before turning it on, it is worth finding out what acks=all promised and what it did not. The same order is stored at two different offsets (positions in the log). Which side wrote the second copy, the broker or the producer? Make a guess before reading on.
Pause and predict acks=all was on, broker 2 died, and order 8812 sits in the log at two offsets. Where did the second copy come from?
A safe setting, and a retry
Here is what happened to one order. Step through it.
Nothing was lost: acks=all did its job, and the follower that took over held every record that had been acknowledged. But the order service sent 8812 before the crash and never heard back. From where it stood, the send might have failed, so it did the right thing and retried. The record had in fact landed, so the retry wrote it a second time.
This is the general shape of the problem from lesson 4.1: across a network, a sender that gets no answer cannot tell “it never arrived” from “it arrived and the answer was lost”. If the sender retries, every message arrives at least once and some may arrive twice (at-least-once); if it never retries, no message arrives twice but some may never arrive (at-most-once). The order service is written in Python on librdkafka, whose defaults are acks=all and retries on, but enable.idempotence=false. The Java client has enabled idempotence by default since Kafka 3.0 (fully in 3.0.1, 3.1.1 and 3.2.0, after a bug); librdkafka has not.
A mental model
A railway parcel office works the same way, on paper.
Partitions, offsets and segments
- Partition
- One ordered, append-only log. A topic is split into several partitions so that it can be stored on, and read by, several machines at once.
- For example: Topic
ordershas six partitions. Order 8812 has keykitchen-41, so it goes to the partition that key hashes to, at the next free offset: 40,117.
- Records stay in sequence only within a partition. Records with the same key land in the same partition and are read in the sequence they were written. Two orders for different kitchens, in different partitions, may be read in either sequence. So choose a key that puts records which must stay in sequence into the same partition: the next lesson does that for the kitchen’s pipeline.
- An offset is a position. It is assigned by the broker when the record is appended, and it says nothing about identity. The retried 8812 got a new offset, which is why nothing downstream could tell it was the same order.
- On disk, a partition is a directory of segment files, each up to 1 GiB by default (
segment.bytes). Writes append to the newest segment. Old data is removed a whole segment at a time.
Consumer groups and rebalancing
The packing screen runs as three processes that share one consumer group name. Kafka gives each partition of orders to exactly one process in the group, so six partitions are split two, two and two. The kitchen screen is a different group, and gets every record too. That is the log of lesson 5.1 at work: groups do not take records from each other.
- Parallelism is capped by partitions. A seventh packing process in the same group would sit idle: there is no partition left to give it.
- A group remembers its position by committing offsets to an internal topic. By default the consumer commits automatically every 5 seconds (
enable.auto.commit=true,auto.commit.interval.ms=5000), so a consumer that crashes can be handed records it had already processed but not yet committed: another source of “twice”. - When a member joins, leaves or stops polling, the group rebalances: partitions are reassigned. In the classic protocol the eager assignors stop everyone while this happens; the cooperative assignor moves only the partitions that must move. In Kafka 4.0 a new consumer group protocol (KIP-848), in which the broker decides the assignment, became generally available; clients opt in with
group.protocol=consumer.
The 12:40 failure in numbers
In the simulator, with three replicas, followers 15 ms behind and the leader dying at order 14:
| Producer settings | Acknowledged, then lost | Stored twice | Time to acknowledge |
|---|---|---|---|
| acks=1 | 3 (orders 11 to 13) | 0 | 0 ms |
| acks=all | 0 | 3 (orders 8 to 10) | 30 ms |
| acks=all, idempotent | 0 | 0 | 30 ms |
acks=1 answers as soon as the leader has the record, so whatever the followers had not copied dies with it. acks=all waits for the followers before it answers: one follower lag (15 ms) for the followers to copy the record, and another 15 ms for the leader to learn that they have it, so 30 ms in all. Nothing acknowledged is lost. But orders 8 to 10 were stored on the followers during that 30 ms wait and never acknowledged, so the producer retried them and they were written again. With idempotence on, the new leader recognises those retries and does not store them again, and the time to acknowledge stays at 30 ms.
How many brokers can fail?
| Replicas, min.insync.replicas | Brokers lost, still accepting acks=all writes | Brokers lost, acknowledged data still safe |
|---|---|---|
| 3, 1 | 2 | depends on the ISR at the time: possibly 0 |
| 3, 2 | 1 | 1 (every acknowledged record is on 2 brokers) |
| 5, 3 | 2 | 2 |
Replication factor 3 with min.insync.replicas 2 is the common production choice: one broker can die, or be restarted for an upgrade, with no lost acknowledgements and no refused writes.
Retention and compaction
A Kafka topic is not drained by reading. It keeps records until a retention rule removes them: by default 7 days (retention.ms), no size limit, deleted a segment at a time. The delivery service now takes about 40,000 orders a day, and each order writes about six events (placed, paid, cooking, packed, picked up, delivered) of about 1 KB:
| Quantity | Working | Result |
|---|---|---|
| Events a day | 40,000 x 6 | 240,000 |
| Data kept, one copy | 240,000 x 7 days x 1 KB | 1.68 GB |
| Data kept, replication factor 3 | 1.68 GB x 3 | 5.04 GB |
| Lunch peak (60% of a day in 2 hours) | 144,000 / 7,200 s | 20 events/s |
These are small numbers for Kafka. The three brokers are there for durability and so that many independent readers can read the log; throughput is never the constraint at this size. Here the six partitions are set by how many packing processes must read in parallel, and the three brokers by how many failures to survive.
Some topics hold state rather than history: the latest menu price per dish, the latest status per order. For those, cleanup.policy=compact makes log compaction keep at least the last record for each key and remove older ones, without reordering what remains. A record with a key and an empty value is a tombstone: a delete, itself removed after delete.retention.ms (1 day by default). A new consumer can read a compacted topic from the start and end up with the current value of every key.
Exactly once, where?
Turning on the idempotent producer is the easy part: set enable.idempotence=true and the duplicates caused by producer retries stop. That leaves duplicates that idempotence cannot see. If a packing screen prints a slip and then crashes before committing its offset, the group rebalances, and the process that takes over its partition reads the same order again from the last committed offset.
Kafka also has transactions: a producer can write to several partitions and commit the consumer’s offsets in one atomic step, and consumers set to isolation.level=read_committed see only committed transactions. For a pipeline that reads from Kafka and writes to Kafka, that is exactly-once processing. The packing screen does not write to Kafka. It prints a slip and updates a database row.
Transactions are the right tool for Kafka-to-Kafka jobs, and the data team’s aggregation job will use them. But the effect the business cares about is a box packed in the physical world, outside any Kafka transaction. The Kafka documentation says it plainly: exactly-once delivery to other systems “generally requires cooperation with such systems”. The unique key in the packing table is that cooperation.
What to remember
- A topic is split into partitions: ordered logs, one leader each, records at offsets. Records stay in sequence only within a partition, so choose a key that keeps records which must stay in sequence together.
- A consumer group splits partitions among its members and remembers its place by committing offsets. More members than partitions sit idle. Rebalances and late commits cause re-reads.
- An acknowledgement means what acks says. acks=all with min.insync.replicas 2 and replication factor 3 survives one broker with nothing acknowledged lost; unclean election trades acknowledged data for uptime.
- A retry after a lost answer writes twice unless the idempotent producer is on. Transactions extend that to read-process-write inside Kafka.
- Outside Kafka, exactly once is at least once plus idempotence at the place the effect happens.
Without the idempotent producer, a retried send is stored as a second record, whatever acks is set to.
In the real world
The simulator follows one partition, one failure and one record per tick. Real clusters add what it leaves out: batching and compression (a request carries many records), follower fetches that bring many records at once, a leader that waits for a whole batch before answering, and timeouts (delivery.timeout.ms, two minutes by default) that decide when a producer finally gives up and reports an error.
A producer waits up to linger.ms to fill a batch of up to batch.size bytes. A longer wait means fuller batches and fewer requests, and slower acknowledgements: the same throughput-versus-latency dial as continuous batching on a GPU. Batching in LLM From First Principles.
Operators also watch two numbers the simulator hides: under-replicated partitions (partitions whose ISR is smaller than their full set of replicas, so one more failure could make them refuse or lose writes) and consumer lag (how far a group is behind the end of the log). In lesson 5.3 the kitchen’s whole pipeline is designed on these internals, and in lesson 6.5 a database’s own log feeds a topic like this one.