Numbers that change decisions
Which numbers should change your design, and how do you get them in two minutes?
How big is “millions of users”?
Someone who wants the kitchen to grow hands over a printed article about a food app that crashed on its launch day. The article’s advice is to plan for millions of users, to shard the database (split it across many machines) from the first day, and to use Cassandra, a database built for that. Otherwise, it says, you will have to rebuild everything when the app becomes popular.
Before choosing Cassandra, find out how many orders the kitchen actually gets. The ledger, where every order is recorded, has the answer: 604 orders in 30 days. That is about 20 a day, and almost all of them are placed between 10 and 11 in the morning.
Now test the article’s advice. Is a million users a lot for one server, or nothing? How many requests a second is that? How much disk? Guess each one before reading on. If you cannot put a number on any of them, you are about to choose a database on the strength of an adjective.
The same week a 78-year-old regular who orders a low-sugar lunch every day phones to say the app “hangs sometimes”. The dashboard says the average order page loads in 180 ms. By that number nothing is wrong, yet a customer who uses the app every day says it hangs.
That leaves two questions, both about numbers. How do you turn “millions” into requests, bytes and machines in two minutes? And how do you measure speed so that the average cannot hide the customer who waits?
Pause and predict A daily customer says the app hangs; the average is 180 ms. Which number would show what he feels?
Adjectives make designs
“Big”, “fast”, “highly available”: each of these words has started an expensive design. None of them can be checked. A number can.
Here is the kitchen today, in numbers. 604 orders in 30 days is about 20 a day. Say each order is about 30 requests from the phone (menu, prices, the order, the confirmation, polling for status). That is about 600 requests a day.
- A day has 86,400 seconds. 600 requests spread over the day is 600 / 86,400 = 0.007 requests per second.
- But they are not spread out. Almost all orders arrive in one hour. 600 requests in 3,600 seconds is 0.17 requests per second at the peak.
- One ordinary server handles hundreds to thousands of simple requests a second. The kitchen uses well under a thousandth of it.
Pause and predict Now take “millions” seriously. 1 million people order once a day, 30 requests each, and lunch makes the peak 10 times the average. About how many requests a second is the peak?
The second problem is the average. 180 ms sounds healthy. But an average blends the fast majority with the slow few, and the slow few are real people.
- Percentile
- Sort every request from fastest to slowest. The 99th percentile (p99) is the time that 99% of requests beat. One request in a hundred is slower.
- For example: p50 = 120 ms, p99 = 2.1 s: half the orders load in 120 ms, but one in a hundred takes over two seconds. A customer who orders every day meets that one regularly.
A mental model
Three letters: SLI, SLO, SLA
Google’s Site Reliability Engineering book gives the vocabulary for non-functional requirements.
| Term | What it is | Example |
|---|---|---|
| SLI, indicator | A carefully defined measurement | Share of order requests that succeed; p99 latency of the order page |
| SLO, objective | A target for an SLI over a window | 99.9% of order requests succeed, over 30 days |
| SLA, agreement | A contract with consequences if the SLO is missed | None yet: the kitchen has no contracts that pay out. An office customer might one day ask for one. |
An SLO implies an error budget: the failure it allows. 99.9% success means 0.1% may fail. Over 30 days that is 0.1% of 43,200 minutes, 43.2 minutes. Spend it on deploys, experiments and bad luck; when it runs out, slow down and fix reliability.
The numbers worth knowing
Estimates need a few reference latencies. These are orders of magnitude from the list Jeff Dean popularised; modern memory and SSDs are faster than the original figures, so use them to compare, not to quote.
| Operation | Roughly | Compared with memory |
|---|---|---|
| Read from main memory | 100 ns | 1x |
| Random read from an SSD | tens to 150 µs | about 1,000x |
| Round trip inside one data centre | 0.5 ms | about 5,000x |
| Seek on a spinning disk | 10 ms | about 100,000x |
| Round trip across an ocean | 150 ms | over 1,000,000x |
The point is in the gaps: each tier is hundreds or thousands of times slower than the one above it. So keep the data you read most often in the fastest tier you can, and count how many trips each request makes to the slower tiers.
Two estimates, worked
When does the kitchen need more? Use the estimator’s assumptions for the “today” preset: each user makes 1 order a day, there are 20 reads per write, each record is 2 KB, the peak is 10 times the average, one server handles 1,000 requests a second and one database node holds 4 TB.
| Decision | Flips when | Working | Daily users |
|---|---|---|---|
| Add a cache | peak reads pass 100 per second | users x 20 / 86,400 x 10 > 100 | ≈ 43,000 |
| Add servers | peak requests pass 1,000 per second | users x 21 / 86,400 x 10 > 1,000 | ≈ 411,000 |
| Shard the data | 5 years of data pass 4 TB | users x 2 KB x 365 x 5 > 4 TB | ≈ 1,100,000 |
So Cassandra answers a problem that starts at about a million daily users, which is 50,000 times today’s 20 orders a day. Long before that, the first change is a cache, then more servers. The estimate does not say “never shard”. It says what to do first, and when.
What does 99.9% allow? The window is 30 days.
| SLO | Allowed to fail | Working | Budget per 30 days |
|---|---|---|---|
| 99% | 1% | 43,200 x 0.01 | 432 min (7.2 h) |
| 99.9% | 0.1% | 43,200 x 0.001 | 43.2 min |
| 99.95% | 0.05% | 43,200 x 0.0005 | 21.6 min |
| 99.99% | 0.01% | 43,200 x 0.0001 | 4.32 min |
Each extra nine divides the budget by ten. A request-based SLO works the same way: at about 18,000 requests a month, 99.9% allows 18 failed requests.
How many nines?
The availability target has to be written down. One goal is that it sounds serious enough to hold the team to a high standard. The other is that it is a promise the team can keep with the server it has.
Four nines does force good engineering: redundant servers, deploys without downtime, alerts that page a person. But 4.32 minutes is less time than it takes one engineer to wake up and open a laptop, and the SRE book makes a sharper point: past a certain level, extra nines are invisible, because the customer’s own phone and network fail more often than the service. Its example is a user on a 99% reliable smartphone, who cannot tell 99.99% from 99.999%.
What to remember
- Turn adjectives into numbers: users to requests per second (divide by 86,400), times a peak factor; records times bytes times years, times copies, for storage.
- An estimate is for deciding which side of a line you are on. Two significant figures are enough.
- Measure latency as percentiles. The average hides the tail, the tail is real customers, and fan-out makes it common.
- An SLI is the measurement, an SLO the target over a window, an SLA the contract. 99.9% over 30 days leaves an error budget of 43.2 minutes; each extra nine divides it by ten.
Turn every adjective into a number, and watch the p99.
In the real world
Real capacity planning starts from measurements: load tests to find what one server really handles, and traffic history to find the real peak factor. The round numbers here (1,000 requests per second per server, a 10x lunch peak) are reasoning aids; replace them with measured values as soon as you have them.
Real SLOs are also written more carefully than here: which requests count (does a refused order count as a failure?), where they are measured (at the server or in the app), and over which window. The observability design (lesson 5.7) returns to how they are measured and alerted on.