Observability

Why does nobody know what happens at 12:45?

Every weekday at 12:45 the order page slows down for a few minutes. Every weekday at 12:45 the on-call phone buzzes with the same page: “CPU above 80% on order-api-2”. By the third week whoever is on call reads it, sighs, and turns the phone face down.

One Saturday at 19:00 a configuration change broke the payment step. Every order failed. The CPU alert stayed silent: a failed order costs a server almost nothing. The error-rate alert would have fired, but it fired every lunchtime too, and it had been muted on Tuesday.

The team found out at 19:40, when a customer phoned the kitchen: forty minutes after the failures began.

Here is what those forty minutes cost. The service promises that 99.9% of orders succeed each month. A month has 43,200 minutes, and a tenth of a percent of that is about forty-three minutes of allowed failure. Nearly all of it went in one evening, while the alarms were busy with lunch.

On Monday the logs for 12:45 held thousands of lines saying “slow request, 3,104 ms”, and not one said why. The dashboards were no better: CPU rose on two servers, the database got busier, the payment service got slower. Every graph moved at 12:45. None of them said which one moved first.

Turning on debug logging everywhere would record more, but the team already logged a great deal and nobody could find the answer in it. What they needed was a way to follow one slow request through every service and see where its time went.

So there are two problems. The alarms ring for the wrong things, and the data cannot answer the one question that matters during an incident: where did this request spend its time?

Pause and predict The CPU alert pages every lunch and stayed silent through a forty-minute payment outage. What should page the on-call engineer?

Watching the machines, not the meals

The alerts watched causes: CPU, memory, a raw error count. A cause is a condition inside the system that may or may not hurt customers. CPU at 85% during the lunch rush is the system doing its job. CPU at 5% during a total payment outage is the system failing while doing almost no work. The alert was wrong in both directions.

Google’s SRE book frames monitoring as two questions: “what’s broken, and why?” The first is a symptom that a customer can feel (orders fail, pages are slow). The second is a cause. Page a human on symptoms; use causes to find the fix. And “every page should be actionable”: a page that is ignored every day trains people to ignore the one that matters.

The data had a different problem: there was plenty of it, but it could not explain why a request was slow. Software produces three kinds of telemetry, and each answers a different question:

The team had metrics for machines, and logs as free text. It had nothing that connected one slow request to the step that made it slow.

A mental model

A lunch-box network that runs on the railway has a way of finding a late box.

Trace and span

A trace is the story of one request. It is made of spans, one per unit of work (a service call, a query, a wait), each with a start time, a duration, a parent and some attributes.

For example: The order request is the root span. Its children are “check session”, “GET menu:today”, “acquire DB connection” and so on. Every span carries the same trace id.

The same tree describes an LLM agent: a production agent’s trace is a distributed trace whose spans are model calls and tool calls.

Add OpenTelemetry to the order API, the payment service and the database client, and write logs as structured records that carry the trace id. Then open one slow request from 12:46.

The order spent 2,870 of its 3,100 ms waiting for a database connection. None of the work itself was slow; the request spent most of its life waiting. Searching the logs by that minute finds the reason: at 12:45 the dispatch job, which runs inside the order API, creates tracking links for 1,500 boxes. For each one it holds a database connection while it waits for the SMS gateway. The pool has 20 connections, so while the job holds all 20, customer requests wait in line for one.

The fix takes an afternoon: release the connection before any network call, and give the job its own small pool. The metrics showed when the slowdown happened, the trace showed where the request spent its time, and the logs explained why. The utilisation of the pool, its saturation (46 waiting) and its errors are what the USE method asks you to watch for every resource; the request rate, errors and duration of each endpoint are the RED method. Both belong on dashboards. Neither should page anyone on its own.

The budget in minutes

The dip is real and worth fixing, because it uses a sixth of the month’s budget. But at 0.56% of the budget a day it cannot break the SLO on its own, so it should open a ticket to be fixed during working hours, and it is no reason to page anyone at lunch. Saturday’s outage spent the budget a thousand times faster than the rate that would last the month; under the 14.4 rule it would have paged within a minute, instead of forty.

Cardinality: the cost hiding in labels

The number of series is the product of the number of values each label can take. http_requests_total labelled by route (20 values), status (5) and kitchen (2,000) is up to 20 x 5 x 2,000 = 200,000 series. Add customer_id with 200,000 values and the possible series become 4 x 10¹⁰. That is cardinality, and it is the most common way to break a metrics system. Prometheus’s own guidance is direct: do not use labels for “user IDs, email addresses, or other unbounded sets of values”. Put the customer id in the log record and on the span, where one event costs one record.

What may wake someone up?

Two goals compete. One is early warning: a signal that trouble is coming before customers feel it. The other is a pager that always means something, so that nobody learns to ignore it.

The wish for early warning is fair. Some causes predict certain harm that an SLO sees too late: a disk that will be full in two hours, a certificate that expires tonight. Those few may page. High CPU at lunch does not predict an outage; it is the normal result of the lunch rush.

What to remember

1. Metrics say when and how much; traces say where the time went; logs say what happened. Connect them with a trace id.
2. Page on symptoms customers feel, measured as an SLI. Causes (CPU, pools, queues) belong on dashboards: RED for services, USE for resources.
3. Burn rate = bad ratio / (1 − SLO). An alert at rate b over window w fires after spending b × w / 30 days of the budget.
4. Multiwindow, multi-burn-rate alerts (14.4 over 1 h, 6 over 6 h, each with a short window) page fast for outages, catch slow leaks, and stop soon after recovery.
5. Keep labels low-cardinality: never a customer id in a metric label.

Page on what customers feel, measured as error-budget burn; put CPU on a dashboard.

In the real world

The simulator assumes steady traffic and evaluates once a minute. Real alerting has to cope with low traffic, where one failed request at night is a huge burn rate (the workbook suggests synthetic traffic or grouping services), with SLIs that combine latency and errors, and with several SLOs per service. Real tracing has to cope with queues and batch jobs, where one request starts work that finishes later, sometimes in several services at once.

The metrics that feed these alerts have to be stored and queried somewhere, millions of samples a minute. That is the design in lesson 5.8.