Consistency, CAP and PACELC
When copies disagree, what exactly can a system promise?
Who gets the last box when the line is down?
The kitchen now cooks in Pune and Mumbai. Each city’s counter screen talks to a copy of the data in its own availability zone, and the two copies keep each other up to date.
It was set up that way after a bad Tuesday in Mumbai. The link to Pune slowed down, every screen waited on it, and the Mumbai counter stopped taking orders for twenty minutes. A counter that stops looked worse than a counter that is wrong for a minute, so each copy now answers at once and sends its changes to the other in the background.
On festival morning there were four boxes of the festival special left. At 12:04 the link between the zones dropped for ninety seconds. Neither copy could hear the other. Pune sold four boxes. Mumbai, still seeing four left, sold four more.
When the link came back, both copies agreed straight away: zero left, eight sold. Four customers had been told yes for boxes that never existed. The accountant would rather the Mumbai screen had said “wait a minute”. On Tuesday, twenty minutes of “wait” had cost a whole lunch.
The Mumbai screen’s log shows something else from those ninety seconds, which the team now calls the blip. For a minute after Pune sold out of dal, Mumbai kept showing it on the menu. Nobody complained about that.
While the link is down, a screen can either always give a correct answer or always give some answer, and it cannot do both. It does not have to make the same choice for the dal and the last box, though. Decide for each before reading on.
Pause and predict While the link is down, what should the Mumbai screen do for the last four boxes, and for the dal on the menu?
Two copies, one cut wire
Once data lives on more than one machine (lesson 3.1), every write has to travel. While it travels, the copies disagree. Most of the time the trip takes a few milliseconds and nobody notices. Sometimes the trip cannot happen at all.
- Network partition
- A fault in which some machines can still run but cannot talk to some others. Messages between the two groups are delayed or lost.
- For example: The link between the Pune and Mumbai zones drops for ninety seconds. Both copies are healthy; neither can hear the other.
Step through the festival-morning blip. Watch what the Mumbai copy knows at the moment the office orders.
Pause and predict Mumbai sold the box from its own copy. During the cut, what else could it have done with the office’s order?
A mental model
Two shops sharing one stock faced the same choice long before anyone had a screen.
What “consistent” can mean
The word is used for several different promises. From strongest to weakest, the ones you will meet in practice:
| Model | The promise | Festival box example |
|---|---|---|
| Linearizable | The system behaves as if there were one copy, and every operation takes effect at one instant between its start and its end. Once any read sees a write, every later read does too. | Once a purchase is confirmed, no screen anywhere can sell that box again. |
| Sequential | Everyone sees operations in one agreed order that respects each client’s own order, but that order may lag real time. | Every screen agrees which order came first, though a screen may be a little behind. |
| Causal | If one operation could have caused another, everyone sees them in that order. Unrelated operations may be seen in different orders. | No screen shows “refund issued” before “order paid”. |
| Session guarantees | Promises to one client: it reads its own writes, and it never sees time go backwards (monotonic reads). Taught in lesson 3.1. | A customer who has paid never sees her own order as unpaid. |
| Eventual | If writes stop, all copies converge to the same value. No promise about what a read returns in the meantime. | After the blip, both copies agree: zero left, eight sold. |
- CAP theorem
- When the network is partitioned, a replicated system must choose: either every request to a working machine gets an answer (available), or every answer behaves as if there were one copy (linearizable). Not both.
- For example: During the blip, the Mumbai copy either sells from its own tally (available, may oversell) or refuses until Pune answers (consistent, some customers wait).
Pause and predict The popular version says “pick two of C, A and P”. Can a replicated system pick C and A and give up P?
Pause and predict The link is healthy. Does a consistent festival box sale from Mumbai cost anything?
- PACELC
- CAP extended to the normal case: if there is a Partition, choose Availability or Consistency; Else, when all is well, choose lower Latency or Consistency.
- For example: With the link healthy, a consistent sale from Mumbai still waits one round trip to Pune; an available one answers locally. You pay for consistency every day, and again during a blip.
What each choice costs, in minutes and milliseconds
Use the lab’s numbers: a round trip of 8 ms between the copies (illustrative; Pune to Mumbai is about 150 km, so light in fibre alone needs about 1.5 ms there and back) and 2 ms to answer from a local copy.
| Quantity | Working | Result |
|---|---|---|
| Consistent sale from Mumbai, healthy link | 2 + 8 | 10 ms |
| Available sale from Mumbai, healthy link | 2 | 2 ms |
| A page making 3 consistent reads one after another | 3 x 8 extra | +24 ms |
| Cuts totalling 90 s in a 30-day month | 90 / 2,592,000 | 0.0035% of the month |
| Mumbai sale availability, consistent choice | 1 - 0.000035 | 99.9965% |
| Share of a 99.9% error budget (43.2 min) | 1.5 / 43.2 | 3.5% |
| Available sales: oversell starts above | one-way lag 4 ms → 1,000 / 4 | 250 orders a second |
Read the table as the owner of the business would. Choosing consistency for festival boxes costs Mumbai customers a few milliseconds every day and, in a month like this one, ninety seconds of “try again”. Choosing availability costs nothing on normal days, and four apologies on festival morning.
The last row is easy to miss. Even with a healthy link, an available design oversells whenever two sales of the last box land closer together than the copies can tell each other. At 12:30 the screens take a few orders a second, far below 250, so on normal days it almost never happens. During a flash sale it would.
Which way should the festival box fail?
Two goals collide here, and each has a bad day on its record: the twenty-minute stop on Tuesday, and the four apologies on festival morning.
Both paths describe real costs. What settles it is that they describe different operations. Tuesday’s outage stopped everything, including the menu, the cart and order tracking, because every read went through the far link. The festival box is the one thing where a wrong answer is worse than a slow one. So each operation gets its own answer.
The case for selling through a cut survives in a better form. Before a festival sale, Pune can hand Mumbai an allotment, say two of the last boxes, which Mumbai may sell on its own with no call to Pune. A partition then only blocks Mumbai once its allotment is gone. Splitting a scarce quantity into pieces each copy owns outright is an old technique (often called escrow); it buys back availability without ever overselling.
What to remember
- “Consistent” has several meanings. Linearizable means one-copy behaviour in real time; causal and session guarantees are weaker and cheaper; eventual only promises that copies converge once writes stop.
- CAP: during a network partition, a replicated system cannot be both linearizable and answer every request. Partitions are not optional, so “pick two of three” is the wrong reading.
- PACELC: even with no partition, stronger consistency costs latency, every day, for every request that must consult a distant copy.
- Choose per operation: strong where a wrong answer costs money or a promise (stock, payments), available where a stale answer is harmless (menus, tracking).
- Scarce quantities can be split into allotments each copy owns, which buys back availability without overselling.
During a partition, let harmless reads answer from the local copy and give every scarce count one home.
In the real world
Real databases expose this choice per operation. DynamoDB reads are eventually consistent unless a request sets ConsistentRead, and a strongly consistent read costs twice as much read capacity. Cassandra takes a consistency level on every query. MongoDB has read and write concerns. PostgreSQL with replicas lets you route a read to the primary or to a replica that may lag.
The lab has two copies and one clean cut. Real partitions are messier: links that are slow rather than dead, partitions that only some machines see, clocks that disagree, and timeouts that must guess. Jepsen’s test reports are a sobering record of databases that promised more than they delivered when the network misbehaved.
The festival stock now has one home. The next question is who decides which machine is that home, and what happens when two machines both believe they are: lesson 3.4.