Diwali night at Nimbu Pay

Three alarms

Diwali, 9:04 p.m. Nimbu Pay’s festival sale opened at 8. You are the engineer on call, and three alarms fire in the same minute.

The first alarm is the refund dedupe job. Every night it reads the day’s refund requests for failed payments, drops the duplicates and sends one file to the bank. The bank stops accepting files at 11:00 p.m. Miss that and the money waits until after the holiday. On a normal day there are 20,000 requests. Tonight there are 600,000. The job started at 9:00, it compares each request with every request before it, and its log says it manages 10 million comparisons a second.

The second alarm is the support bot. 120 messages a second are reaching it. Each reply keeps a connection to the model open for 4 seconds on average, and the bot has room for 400 open connections. Its p99 reply time is 31 seconds and climbing.

The third alarm is a message from the product manager who owns support: customers are posting screenshots of the slow bot, and can you fix it now? It is the loudest of the three.

You can work on one thing at a time. Before you open anything, decide which.

The chapters

1. Three alarms
2. The refund file
3. The bot is drowning
4. The bill
5. Before it ships
6. The morning after