Queues, retries and idempotency
How do you do slow work later without doing it twice?
Why did one customer’s phone buzz twice?
The delivery service now cooks with 2,000 home cooks in three cities, for about 200,000 customers. At the 12:30 peak about twelve orders arrive every second.
At 12:31 one Monday a regular customer ordered his low-sugar thali. The spinner turned for ten seconds. Then “Confirmed” appeared, and his phone buzzed twice with the same message. He rang the kitchen to ask whether he had ordered two lunches. He had not.
At the same moment the menu page stopped loading for everyone.
The order code tells the story. After saving the order it called the SMS provider and waited for an answer, inside the customer’s request. The provider usually answers in two seconds. At lunch that Monday it took ten. The code gave up after five and tried again. The first attempt had gone through, so both messages arrived, and the business paid for both.
The waiting did the other damage. Every request held a server thread while it waited for the SMS. Do the sum before reading on: twelve orders a second, each holding a thread for ten seconds, is 120 threads. The three servers have 96. The menu requests queued behind the orders.
The customer needs the order saved. The SMS can follow. But if nobody waits for it, how do you know it was sent? And if you send it again to be sure, how do you stop it arriving twice?
Pause and predict Twelve orders a second, each holding a thread for ten seconds, need 120 threads, and the servers have 96. What frees the threads without losing a single SMS?
A timeout tells you nothing
Step through Monday’s order. Watch where the thread is held and what the server knows after its timeout.
Pause and predict The server gave up on the SMS provider after 5 seconds. What does it know about that first message?
Two separate mistakes are tangled together here.
- Slow work on the request path. The customer needs to know the order is saved. They do not need to wait for the SMS. Anything the reply does not depend on can happen later (lesson 1.5’s click counting was the first example).
- Retrying something that is unsafe to repeat. The server could not tell a lost message from a sent one, and it sent again anyway.
The thread arithmetic has a name. Little’s law says the number of things inside any stable system equals the rate they arrive times how long each one stays: L = λW. Twelve orders a second (λ) that each stay ten seconds (W) means 120 orders inside the servers at once. It holds for requests and threads, jobs and workers, and boxes on the road.
A mental model
A kitchen office already handles work that should not hold anyone up.
- At-least-once delivery
- A queue that may hand the same job to a worker more than once, but never loses it. Almost every real queue offers this.
- For example: A worker takes the SMS job, sends the message, and crashes before deleting the job. After the visibility timeout another worker takes it and sends it again.
- Visibility timeout
- When a worker receives a job, the queue hides it from others for a set time instead of deleting it. If the worker deletes it in time, it is gone. If not, it becomes visible again.
- For example: Amazon SQS hides a received message for 30 seconds by default, adjustable up to 12 hours. Set it longer than the work takes, or slow jobs will be handed out twice.
Little’s law, before and after
| Quantity | Working | Result |
|---|---|---|
| Threads held by orders, SMS on the request, provider at 10 s | 12/s x 10 s | 120 (servers have 96) |
| The same on a normal day, provider at 2 s | 12/s x 2 s | 24: why it worked until Monday |
| Threads held by orders, SMS on a queue, 60 ms reply | 12/s x 0.06 s | 0.72 |
| SMS workers busy, provider at 2 s | 12/s x 2 s | 24 of 40 (60%) |
| SMS throughput with 40 workers, provider at 10 s | 40 / 10 s | 4/s, while 12/s arrive |
| Queue growth during a 30-minute slow spell | (12 - 4)/s x 1,800 s | 14,400 jobs |
| Draining it once the provider recovers (20/s with 40 workers) | 14,400 / 20 | 720 s = 12 min |
| Longest possible backoff before the 8th attempt | 1 + 2 + 4 + 8 + 16 + 30 + 30 s | 91 s (average 45.5 s) |
The queue moved the slowness off the customer, but it did not make the provider faster. When the provider is slow, either the workers grow (120 to keep up at 10 s), or the queue grows and messages arrive late. A queue turns an outage into a delay; it is only useful if someone is watching how long the delay has become.
Lose one, or send two?
With a queue in place, the confirmation needs a delivery promise. Two goals compete: never alarming a customer with a second message, and never leaving a customer without one.
The numbers come from the queue lab in Try it, with its defaults and the outage on. A duplicate has a real cost for some customers, and for some messages the first path wins: a one-time login code is useless a minute late, so it should be retried briefly and then abandoned. For an order confirmation, losing a third of them during every provider blip is worse than a minute’s delay.
What to remember
- Do on the request only what the reply needs. Put slow or flaky work (messages, emails, thumbnails) on a queue for workers.
- A timeout means “I do not know”. Queues deliver at least once. Plan for every job running twice.
- Retry only what might succeed, at one layer, with capped exponential backoff and full jitter, inside a retry budget.
- Make repeats harmless with an idempotency key that names the intent; check it where the effect happens, or in the same transaction as your own write.
- Little’s law, L = λW, sizes threads, workers and pools, and tells you how fast a queue grows when workers fall behind. Send jobs that never succeed to a watched dead-letter queue.
Reply when the order is saved; send the rest from a queue, with a key that names the intent.
In the real world
The simulator models one queue and one provider. Real systems add details that matter: workers extend the visibility timeout while a long job runs (a heartbeat), batches of messages are received and deleted together to save calls, priorities keep a login code from waiting behind a marketing blast, and the alarm that matters is the age of the oldest message in the queue, more than the length of the queue.
The pattern returns often. The notification system (lesson 4.5) is this queue grown into a product, with several channels and provider rate limits. Kafka (lesson 5.2) offers exactly-once inside its own log by the same trick: at-least-once plus recognising repeats. And lesson 6.5 closes the gap left open here, saving the order and enqueuing its job as one write.
Inside one program, the outgoing tray is a plain first-in, first-out queue, covered in queues and deques in Coding Patterns.