Design a notification system

Why did one box send three messages?

At 12:52 one afternoon a customer’s phone buzzed three times in a minute. “Your dabba has left. Track it here.” Three identical texts.

He rang the kitchen. He orders one low-sugar box a day and plans his insulin around it, and he wanted to know whether three boxes were coming. The kitchen screen showed one order, one box, one rider already on the road.

The worker’s log had the cause. In lesson 4.1 texts moved off the order path into a queue, with a worker that retries failures with backoff. That lunch the SMS gateway was slow. It took twelve seconds to answer some sends, and the worker gave up after ten and tried again. The first send had worked. And because the call took so long, the worker missed the queue’s acknowledgement deadline, so the queue handed the same job to a second worker. Three sends, three texts.

At three o’clock the festival offer went out to all 200,000 customers through the same queue. A student tried to log in to order dinner. His one-time password arrived seventeen minutes later, after it had expired. Work out why before reading on: the gateway accepts 200 sends a second, and the offers were queued first.

The kitchen pays for every one of those texts, the copies and the offers that pushed the passwords back. The cheap answer is to put everything on push notifications, which cost nothing per message. That would fix neither problem. The worker would still not know which messages it had already sent, and it would still send every message strictly in the order it was queued.

Retries were right; lesson 4.1 showed that every network call can fail. But a retry that cannot tell “it failed” from “I did not hear back” turns one message into three. What the kitchen needs is a notification system: a service every part of the business calls, that sends each notification once, sends the urgent ones first, and still works when the providers do not.

Pause and predict The gateway took twelve seconds and the worker gave up at ten and retried. What stops the second and third texts?

What must never go wrong?

Start with the context: who receives notifications, our one system, and the outside services that carry the last mile.

The three texts give the first requirement: no copies. The late password gives the second: urgent messages are sent quickly whatever else is queued. The third comes from the kitchen’s accounts: a notification that fails must never disappear without someone being told.

Core entities, in two minutes

A Notification (one per event and channel, with a status), a Template, a person’s Preferences and push Devices, and the Attempts and receipts each send produces.

The estimate that changes the design

Twelve messages a second is nothing for our servers. The provider’s limit is what matters: providers cap how fast you may send, and a campaign occupies that cap for 16.7 minutes. This estimate decides the shape of the design: urgent messages cannot share a queue with a campaign, however fast the workers are, because the provider’s limit is the bottleneck.

The obvious version, and how it breaks

The first version is lesson 4.1’s design applied to texts: every part of the business puts an SMS job on one queue, and one kind of worker sends them, retrying any failure with backoff and jitter.

Pause and predict The gateway takes 12 seconds to answer, but it does deliver the text. The worker gives up after 10 seconds and retries. How many texts reach the customer?

Run it. The defaults are the first version: retries on, no check for copies, one queue for everything, and a 20,000-message offer. Then change one switch at a time and watch which number moves.

Three different failures are hiding in one pipeline:

• Queue redelivery. An at-least-once queue hands a job out again if the worker does not acknowledge it in time. A slow send is enough. With nothing to recognise the second copy, it is sent again.
• Unknown outcomes. A timeout does not mean the message failed. Retrying it sends a second copy whenever the first had in fact gone.
• One queue for everything. Every message shares the provider’s limit, and the queue is first in, first out, so an OTP requested at 15:01 waits behind 200,000 offers.

Turning retries off does not fix it. The duplicates from timeouts disappear, but a send that really failed is never tried again, and nobody is told that the message never arrived.

A mental model

Unknown outcome

A call that timed out or lost its connection may have done its work or not. Retrying it is safe only if the other side can recognise the repeat.

For example: The gateway took the text, the network delivered it, and the reply was lost. The worker sees a timeout; the customer sees a message.

Resend an unknown, or not?

One SMS provider offers a status lookup. The second, kept for failover, offers no lookup and ignores any key it is sent. When a send to it times out, what should the worker do? Two goals pull against each other: no message lost, and no message repeated.

Both are right about different messages. The cost of each mistake depends on the category, so the policy should too. A duplicate OTP is harmless (same code, short expiry); a lost one locks a customer out. A duplicate “your dabba has left” frightened a customer; a lost one costs little, because the tracking page still works. A duplicate offer is pure waste.

What to remember

1. Callers report events; the notification service decides channel, template and timing, and answers 202 before sending.
2. A dedupe row per (event, channel), claimed with one atomic insert before sending, stops redelivered copies and caller retries.
3. A timeout is an unknown outcome, not a failure. Ask the provider if you can; otherwise choose the resend policy per category from the cost of a duplicate versus a loss.
4. Separate lanes, and a reserved share of each provider’s limit, keep a campaign from delaying OTPs. Size workers for a slow provider (Little’s law).
5. Nothing disappears silently: retries, then a dead-letter queue that pages a person, and receipts that record what really happened.

Record the send before it leaves, and treat a timeout as unknown, never as failed.

In the real world

Real notification platforms add what our toy leaves out: per-user frequency caps (no more than so many offers a week), digests that merge ten small updates into one, scheduling in each person’s time zone and quiet hours, several providers per channel chosen by price and health, and regulatory rules on marketing messages that differ by country and channel. Many teams buy this as a service; the design questions inside it are the ones you met here.

Our simulator draws send outcomes at fixed, illustrative rates and treats the provider limit as a hard cap per second. Real providers fail in bursts, slow down before they fail, and report receipts minutes late or not at all, which is why the receipt endpoint never moves a status backwards.