Distributed job scheduler

Why did the menu go out twice on Monday and not at all on Tuesday?

Every morning at 6:00 a job builds the day’s menu and sends it to a thousand subscribers. For a year it ran from cron on one small server. Then that server was rebooted for an update at 5:58 one morning, and nobody got a menu.

The fix was a second server. The first runs cron; the second listens for its heartbeat and, after three minutes of silence, starts running cron itself.

On Monday a switch between the two servers failed for ten minutes. Each server stopped hearing the other, and each decided it was the one in charge. At 6:00 both sent the menu. A thousand customers got two messages, and the SMS bill was for two thousand.

On Tuesday the first server went down at 5:59. Before reading on, work out what the second one did. It promoted itself at 6:02, as designed. Its cron looked at the clock, saw 6:02, and waited for tomorrow’s 6:00. On Monday two servers sent the menu, and on Tuesday none did.

A Kubernetes CronJob looks like the answer, and the job itself is one line in a file. The hard part is a promise that sounds simple: every scheduled run happens once, close to its time, even when machines die, clocks jump and the network splits. No single mechanism keeps that promise; it takes three pieces working together.

One scheduler, a run record, idempotent jobs

Every piece here comes from a core lesson: one leader with a fence (lesson 3.4), retries with an idempotency key (lesson 4.1), and deciding failure in advance (lesson 7.1). What follows is how they combine into a scheduler.

The three pieces

• One scheduler at a time. The scheduler acts only while it holds a lease, and every write it makes carries a fencing token, so a paused old leader is refused. A heartbeat between two servers cannot do this, because silence looks the same whether the other server crashed or the cable between them was cut.
• At least once, from a durable record. Firing means inserting a run record keyed by the scheduled time, with a condition that it does not exist yet. A new leader scans for times that are due and have no record, so a run missed during a takeover is fired late instead of never. A worker that dies leaves its claim unfinished, and another worker retries it.
• Idempotent jobs. Running at least once means a job will sometimes run twice. Each side effect carries the run id as its idempotency key (here, run id plus subscriber), so a retry skips what already happened.

Durable workflow

A multi-step job whose progress is recorded after each step, so a crash resumes after the last completed step instead of starting over.

For example: Build the menu, send 1,000 messages, mark the day done. The worker dies after 600 sends; the retry starts at send 601.

How much of this does the menu job need?

You could use every piece the lab offers, but each one has a cost.

For a job of two cheap steps, rework is a few hundred key lookups. For a payout job of five steps across a bank and a ledger, restarting from the top is expensive and risky, and that is where durable steps earn their keep.

What to remember

1. “Exactly once, on time” is three pieces: one scheduler (a fenced lease), at least once (a durable run record per scheduled time, retried), and idempotent jobs.
2. A heartbeat between two boxes cannot elect a leader: silence looks the same whether the other box crashed or the cable between them was cut.
3. Fire by inserting the run record for that time; a second claim for the same time fails, and a new leader catches up times that are due with no record.
4. Durable steps turn a retry from “start again” into “continue”, which matters when steps are slow, costly or touch money.

Claim each scheduled run in a durable record before doing it, and make every step safe to repeat.

In the real world

The toy has one job, one worker crash and a clean clock model. Real schedulers also spread jobs across many workers, bound how many run at once, decide what to do with a run that is hours late (run it, skip it, or run only the latest), and record every attempt so someone can answer “did it run on the 14th?” without guessing.

The run record is the same idea as the hold in lesson 6.3 and the inbox in lesson 6.5: write down the claim first, in one place, with a rule that it can be claimed only once.