Sagas and event sourcing

Who gives the customer her money back?

At 12:20 a customer orders a dal and rice box on the way to her shift, with a twenty-percent coupon. Since lesson 6.4 the order touches four services, each with its own database: the kitchen reserves a cooking slot, the coupon service marks the coupon used, the payments service takes ₹240 by UPI, and the rider service finds someone to carry the box.

The first three steps succeed, and her phone says “Confirmed”. The rider who accepted the pickup gets a puncture and cancels. For ten minutes the rider service looks for anyone else near her hospital, finds nobody, and gives up.

The order code catches the error and calls rollback(). The call does nothing, because there is nothing left for it to roll back: the payment committed in the payments database two minutes earlier, the coupon service has already written “used”, and the kitchen’s slot is long reserved.

Count what is wrong before reading on. The bank shows ₹240 taken from the customer. The books show no order. The coupon is spent. Her dal is in a box on the counter, and nobody is coming for it.

Lesson 6.4 looked for one transaction around the whole order and found that two-phase commit holds locks across services and blocks when its coordinator dies. The outbox from lesson 6.5 makes sure each step is announced. Neither says what to do when step four fails after steps one to three have succeeded.

Each of the three committed steps needs a way back of its own, and nobody wrote one. The order has become a business process spread over four databases and ten minutes, and that needs a different tool.

Pause and predict Step four failed after the first three steps had committed, and rollback() does nothing. What does each committed step need so that the order can be cancelled cleanly?

A rollback cannot cross a service boundary

Inside one database, a transaction that fails is rolled back: the write-ahead log (lesson 2.2) lets the database erase every change it made, and other transactions never saw them (lesson 2.3). That promise ends at the database’s edge.

This order is four local transactions in four databases. Each one committed and released its locks as soon as it finished, which is exactly what keeps the services fast and independent. Once the payment has committed, no other service can reach into the payments database and undo it.

• Two-phase commit would keep all four open until the end, holding locks for ten minutes of rider search, and blocking if the coordinator dies (lesson 6.4).
• Doing nothing leaves ₹240 taken, a coupon spent and a slot held, for an order that will never arrive.
• Fixing it by hand works for one order a week. At 40,000 orders a day, a 2% failure rate is 800 orders a day to fix by hand.

Pause and predict The ₹240 payment committed two minutes ago in the payments database. How does the order service undo it?

A mental model

Saga

A long business process split into a sequence of local transactions, each in one service, where every step that might need undoing has a compensating action.

For example: Reserve a slot, apply the coupon, take payment, assign a rider. If the rider step fails: refund, give the coupon back, release the slot.

Compensating action

A new transaction that cancels the business effect of an earlier one. It does not restore the old bytes; it records that the effect was reversed.

For example: A refund of ₹240 is a new row in the payments ledger. The original payment row stays, as a paper ledger keeps a correcting entry below the original line.

Two ways to run one

• Orchestration. One component, the orchestrator, tells each service what to do and waits for the reply. It keeps the saga’s state (current step, status) in its own table, so it can resume after a crash. The whole process is readable in one place.
• Choreography. No one is in charge. Each service does its step and publishes an event; the next service reacts to that event. Failure is an event too, and each earlier service reacts by undoing its own step.

Either way, every step must update its database and send its message together, which is the outbox from lesson 6.5, and every receiver must handle a message arriving twice (lesson 4.1).

Counting the cost

The message volume is small for any broker (lesson 5.2). What costs real effort is the code: three compensations, each idempotent, each retried until it succeeds, each tested. And the ten minutes of rider search, rather than the milliseconds of messages, set how long an order stays pending.

Event sourcing: keep the facts, derive the state

The saga’s steps are a list of facts: placed, slot reserved, coupon applied, paid, rider failed, refunded. Event sourcing makes such a list the system of record: instead of storing the order’s current row and overwriting it, you append each event and compute the current state by replaying them (lesson 2.2 did the same for a database page). In the lab in Break the order, “The order as events” shows it: move the snapshot slider, and the rebuilt state stays the same while the number of events replayed changes.

An order has a handful of events and never needs a snapshot. An account that lives for years does. And nobody can ask an event store “which orders are pending in Kothrud?” cheaply, so event-sourced systems also build read models: tables shaped for each question, kept up to date by consuming the events. Separating the model you write from the models you read is CQRS, and the price is that read models lag the events, which is the replica-lag problem of lesson 3.1 appearing in a new place.

Who runs the order saga?

The compensations now exist. The open question is what calls them, and two goals compete: one place to see and audit every order, or teams that each react on their own with no extra component to run.

Choreography has fewer moving parts, and for simple reactions it is the better fit: when an order is delivered, the feedback SMS and the cook’s earnings can each listen for “delivered” with no coordinator at all. But the order saga has four steps, money in the middle, and a strict undo order, and with choreography the process exists only as the sum of four services’ event handlers. When a fifth step arrives, nobody can see the whole thing to check it.

What to remember

1. Across services there is no rollback. A saga is a sequence of local transactions with a compensating action for each step that may need undoing.
2. Compensations are new transactions (a refund, a coupon given back), run newest first, retried until they succeed, and idempotent, because a timed-out call may have worked.
3. Sagas have no isolation. Use pending states (a semantic lock) so nothing treats a half-finished order as final, and put steps that cannot be undone as late as possible.
4. Orchestration keeps the process in one readable place; choreography needs no coordinator, but nobody has a view of the whole process. Long processes with money suit the first; simple reactions suit the second.
5. Event sourcing stores the facts and derives state by replay, with snapshots for long-lived entities and CQRS read models for queries, which lag the events.

For every step that commits, write its compensation before the first order runs.

In the real world

The lab’s saga runs four fixed steps, delivers every message once and never crashes the orchestrator. Real sagas have to handle what makes them hard: the orchestrator’s state lives in a database and every step is resumed after a restart; messages arrive twice or late; a compensation can fail for hours (the bank is down) and must be retried until it succeeds, with a person alerted if it never does; and steps change between versions while old sagas are still running.

Fowler’s article has a warning that matters for every event-sourced system: replaying events must not resend the SMS or charge the card again. Calls to the outside world go through a gateway that is switched off, or remembers its answers, during a replay. The payment system in lesson 6.7 puts the outbox, the saga and an event-sourced ledger together, and the digital wallet side quest builds a ledger on event sourcing end to end.