Going multi-region

Why is Delhi slower after we moved closer?

The delivery service opened in Delhi in October: 300 home cooks, and by the second week about a fifth of all orders.

Every server and every database still ran in the cloud region near Mumbai, where they had been put for Pune in lesson 0.1. From a phone in Delhi, the menu page took about three times as long as it did in Pune. The Delhi cooks open the kitchen screen and wait, and they think the app is cheap.

The obvious move is to start the order service in a cloud region near Delhi, so Delhi phones talk to a server in their own city. The next morning the Delhi menu page was slower than before. The server is closer. How did it get slower?

Two other questions came up at the same time. One is the standard advice: go active-active everywhere, with every region taking every write, because that is what large companies do. The other comes from the accounts. The month before, the payment provider had gone down for an afternoon, and matching orders by hand took a week. If the whole Mumbai region went down at lunch, which orders would be lost?

Until now there was one database, and every request went to it. With two regions, someone has to decide which region holds the data and which one accepts writes.

Moving servers is easy. Moving data is harder, and where the data lives decides where the work must happen, how far every query travels, and what is lost if a region goes down. Before reading on, count the trips a Delhi page makes with the server in Delhi and the database in Mumbai.

Pause and predict The order service moved to Delhi and the Delhi page got slower. Why?

Close servers, far data

Loading a page takes many round trips. In lesson 0.1 you counted them: the phone makes several calls, and behind each call the server asks the database several questions, one after another.

For a Delhi customer with everything in Mumbai, the 4 calls from the phone each cross the country. The move kept those 4 calls in Delhi and sent the 8 database queries across the country instead.

Pause and predict Say 12 ms from a phone to a region in its own city, 28 ms between the two regions, 1 ms for a query inside one region. A page is 4 calls from the phone and 8 database queries. Which is faster for a Delhi customer?

So the choice is where the data lives, and how many copies of it there are. The servers follow.

A mental model

A caterer who opens in a second city learns the same rule.

Region and availability zone

A region is a cloud provider’s location in one metropolitan area, made of several availability zones: separate data centres with their own power and cooling, a few milliseconds apart.

For example: Since lesson 1.5 the service has run in two zones of one region, so losing a building costs nothing. Losing the whole region costs everything.

The shapes of two regions

• Active-passive. One region serves everyone and takes every write; the other holds an asynchronous copy and is promoted only if the first fails. Simple and familiar, and Delhi still crosses the country on every call.
• Active-active, write partitioned. Both regions serve traffic. Each customer, kitchen and order has a home region where its writes go, with an asynchronous copy in the other. No two regions write the same row, so there are no conflicts to merge.
• Active-active, write anywhere. Any region accepts any write and copies it to the others. Fast, and every concurrent update to one row becomes a conflict to resolve (lesson 3.1’s multi-leader replication).
• A global consensus store. One database across three or more regions; a write commits when a majority of regions have it (lesson 3.4). Nothing acknowledged is lost when a region dies, and every write waits for a round trip to another region.

Two numbers describe what a region failure costs. The recovery point objective (RPO) is how much recent data you can lose; the recovery time objective (RTO) is how long you can be down. Both are business choices, set before the failure, and both cost money to shrink.

Two cities in numbers

The simulator’s defaults: 40,000 orders a day, a fifth of them in Delhi, 3 orders a second at the lunch peak, a 28 ms round trip between regions and 2 s of replication lag.

• The floor. Delhi and Mumbai are about 1,150 km apart in a straight line. Light in fibre covers about 200 km a millisecond, so no round trip between them can be under about 11.5 ms. Real paths are longer; the 28 ms here is illustrative.
• Lost orders. An asynchronous copy can lose the writes it had not received yet: 3 orders a second x 2 s of lag = 6. At 30 s of lag, 90. The one-region design falls back to its last snapshot: with hourly snapshots, up to 3 x 3,600 = 10,800 orders if the whole hour was a lunch peak.
• Bytes. Copying each order once to the other region: 40,000 x 4 KB x 30 = 4.8 GB a month. Pulling query results across the country instead: 8,000 Delhi orders x 10 pages x 8 queries x 2 KB x 30 = 38.4 GB. Moving results is eight times the traffic of moving the data once.

One writer, or one per city?

Two goals compete. One is a Delhi page as fast as a Pune page. The other is accounts with as few ways to go wrong as possible. Each goal has a design that serves it best.

One writer is simpler, and for a business whose second city is small, active-passive is a sound choice. What decides it here is the shape of the data: almost every order is local to one city, so partitioning by city creates almost no cross-region work, and Delhi’s cooks stop waiting. The accounts stay safe as well: the few things that are truly global (a customer who moves city, the coupon budget) keep one home and are written there.

What to remember

1. Put compute in the same region as its data. Moving servers away from their database multiplies distance by the number of queries.
2. A second region buys latency for far customers and survival of a region loss. Two zones in one region already survive most disasters.
3. Active-passive keeps one writer; home region per customer partitions writes so regions never conflict; write anywhere needs data that merges; a consensus store loses nothing and pays a round trip on every write.
4. RPO is the data you can lose (an asynchronous copy loses its lag); RTO is the time you can be down. Choose both before the failure, and pay for them.
5. Move data once, in bulk and asynchronously; never pull it across regions query by query. Law can also decide where data lives.

Put the compute next to the data, and move the data once, in bulk, instead of query by query.

In the real world

The simulator counts round trips and multiplies by illustrative times. Real deployments add what it leaves out: a global load balancer or DNS that sends each customer to their home region and stops sending them to a dead one, health checks that must tell a slow region from a dead one, failover runbooks practised before they are needed, and the cost of keeping two of everything patched and in step.

The copy is the easy part. The hard part is deciding, at 12:40 with half the dashboards red, that the region is truly lost and that a few seconds of orders are a price worth paying to bring everyone back. Write that decision down in advance, and practise it.