Design metrics and monitoring
How do you store and query millions of measurements a minute?
Which number goes in the partner’s report?
Twenty partner kitchens now send their orders through the delivery service’s API. Each has signed the same promise: 99% of order requests answered within 300 milliseconds, checked minute by minute through the lunch hour.
Every Monday each partner gets a report. It was built in an afternoon. Each of the six app servers works out its own p99 once a minute and writes one row to a metrics table in PostgreSQL. The report averages the six rows.
One Tuesday the largest partner sent back a screenshot of its own logs. Every day, the report shows 437 milliseconds for the minute at 12:45. The partner’s logs say more than a second, and the partner wants a refund for every bad minute. Which number is true?
The raw request log for one minute at 12:45 settles it. One app server shares its machine with the kitchens’ label-printing job, which starts at 12:45. One in ten of that server’s requests waits about a second. The other five servers are fine. Across all 36,000 requests in that minute the real p99 was 1,096 ms. The average of the six servers’ p99s, 437 ms, did not describe the requests of that minute at all.
Two quick fixes come to mind. Keeping every request in a search index gives exact percentiles whenever anyone asks, at a cost per request. Tagging every number with the kitchen lets any complaint be checked in seconds, at a cost per kitchen. Neither fixes the actual mistake, which was in how the six servers’ numbers were combined.
The same mistake is easy to make on paper. If six cooks each report their single slowest plate of the day, the head of the kitchen still cannot work out how long the slowest plate in every hundred took across the whole kitchen. Before reading on, guess: what should each cook, and each server, report instead?
Pause and predict Six servers each report a p99, and the report averages them to 437 ms. The real p99 was 1,096 ms. What should each server send instead?
What must never go wrong?
One rule comes first: a latency number sent to a partner must be a true statement about requests, to within a stated error, or it must not be sent.
Core entities, in two minutes
A series is one metric name plus one exact set of labels, such as order_latency_bucket{route="/v1/orders", le="0.5"}. A sample is a timestamp and a number in a series. A rule turns series into new series or into alerts; an alert goes to a person. Everything else is built from these four.
The estimate that changes the design
40 targets with about 2,500 series each is 100,000 series. Scraped every 15 seconds that is 6,667 samples a second, 400,000 a minute.
Pause and predict Keep 15 days of those samples at 2 bytes each after compression. How much disk is that?
The obvious version, and how it lies
The first design has a hidden assumption: that a percentile of the whole can be built from percentiles of the parts. At 12:45, with 10% of server 1’s requests stuck, the average of the six p99s is 437 ms and the real p99 is 1,096 ms: the report understates the bad minute by more than half.
Pause and predict Now only 2% of server 1’s requests are stuck. Is the average of the six p99s too high or too low?
The other obvious fix, keeping every request, is correct but expensive: 600 requests a second is 51.8 million rows a day for one metric on one route. Lesson 2.5 showed what that stream does to an indexed table.
A mental model
- Histogram (as a metric)
- A latency measurement kept as counts in fixed ranges (“up to 50 ms”, “up to 100 ms”, …) plus a total count and sum. Counts from many servers can be added bucket by bucket, and any percentile can be estimated from the added counts.
- For example: Server 1 at 12:45, with the default buckets: 5,401 requests up to 250 ms, 9 more up to 500 ms, 152 more up to 1 s, 436 more up to 2.5 s, and 2 slower. Add the other five servers’ counts: 35,562 of the 36,000 requests finished within 1 s, so request number 35,640 (the 99th percentile) lies in the 1 to 2.5 s bucket.
Pause and predict With the client library’s default buckets (5 ms up to 10 s), the 12:45 p99 is known only to lie between 1 and 2.5 seconds. How far off is the estimate from the true 1,096 ms?
Should every number carry the kitchen?
Two goals compete. One is to answer any kitchen’s complaint in seconds, from the same dashboard. The other is to keep the number of series bounded, so the store stays one machine per zone.
The need is real, and for twenty contracted partners the label is exactly right. For 2,000 home kitchens it multiplies every cost of the system by fifty for a question that logs answer well enough. A value with thousands of possibilities, such as a kitchen id, belongs in the request logs, which are built to store one record per request.
What to remember
- Percentiles do not add up. Never average, sum or max per-server p99s; send bucket counts, add them, then take the percentile.
- A histogram’s estimate is only as precise as the bucket around the value you are estimating. Put a bound at every SLO threshold, or use exponential (native) buckets.
- A time-series store is an LSM tree for numbers: WAL and memory, then immutable compressed blocks. Delta-of-delta timestamps and XOR values bring a sample to 1 to 2 bytes.
- Cost scales with the number of series, which is the product of the number of values each label can take. Keep labels bounded; send unbounded detail to logs and traces.
- Pull, so that a target that stops answering is noticed (
up = 0); push only for batch jobs that end. Run the metrics system in pairs, and have something outside it check that alerts can still be sent.
Send bucket counts, never percentiles: counts add up across servers and percentiles do not.
In the real world
Real systems at this scale add what this design names but does not build: sharding scrapers by target as series grow, a horizontally scalable long-term store in front of object storage (open-source projects such as Thanos, Cortex and Mimir take this role for Prometheus), query caching, and per-tenant series limits. Facebook’s Gorilla kept 26 hours in memory in front of a disk-based store; the same split applies here: recent data compressed and quick to reach, older data cheap and coarse.
The partner reports now come from merged histograms with a bucket at 300 ms. The label-printing job moved to its own machine, and the 12:45 bump disappeared from the graph.