Design a payment system

Why was one lunch paid for twice?

Payments now run through a payment service provider, a company that talks to the banks for the delivery service. At 12:40 one Saturday a student pays ₹180 for a thali from his hostel room. The screen spins for ten seconds and says “Payment failed. Try again.”

He taps Pay again. This time it works. Then his phone buzzes twice: two debits of ₹180 for one lunch.

Across town another customer paid at 12:44. Her bank debited her, but her order still says “pending” at two o’clock, and the kitchen never cooked it.

On Monday the provider’s settlement arrives: everything it collected on Saturday, paid into the business’s bank account in one lump. Compared with the orders table, the lunch hour is off by ₹8,670, and the bank holds more than the recorded sales. Extra money may sound like good news, but it means the business’s own records and the bank disagree, and nobody can explain the difference.

The payment code is short: it calls the provider, waits for the answer, and sets paid to true. It has no plan for the case where no answer comes back. When a call times out, the money may have moved or it may not have. The code treated the timeout as a failure, told the student to try again, and he paid twice. For the second customer, the provider’s confirmation never reached the server, so her order stayed pending.

A shop counter has two old habits for exactly this: write the receipt number before taking the money, and count the cash box against the receipt book every night. The payment code does neither. Before reading on, decide which of the two habits would have saved the student’s second ₹180.

Pause and predict The student’s first call to the provider timed out, the code reported “failed”, and his second tap charged him again. When he first taps Pay, what should the app record before it calls the provider, so that a second tap cannot charge him twice?

What must never go wrong?

Start with who handles the money and which parts are outside our control. We do not run the banks or the provider. The only records we control are our own payment records and our own ledger.

The accountant’s rule is the first requirement: the business never keeps money it did not earn, and never loses track of money it did.

Core entities, in two minutes

A Payment intent (one attempt to collect one amount for one order, with a status), its idempotency key, Journal entries made of postings to accounts, provider events we have received, and settlement lines from the provider’s daily file.

The estimate that changes the design

What the estimate shows: volume is not the problem. At this scale the whole ledger is a few gigabytes a year, which fits in one PostgreSQL database, with one local transaction for each status change and its postings. The hard part is uncertainty: calls that time out, messages that arrive twice or never, and a bank that keeps its own book. (At 1,000 payments a second the ledger needs partitioning by account and time, and balances must be kept as running totals instead of sums.)

The obvious version, and how it breaks

The first version does the payment inside the order request. The order service calls the provider, waits up to ten seconds, and sets orders.paid = true on success. On any error the phone says “failed, try again”.

Pause and predict The provider call times out after ten seconds. The customer’s bank may or may not have debited him. What should the phone tell him?

Three things break it. None of them is a wrong line of code; each is a case the design never considered:

• A timeout does not mean the payment failed. The provider may have reached the customer’s bank, which debited him, and only the answer was lost. Treating the unknown as “failed” invites a second tap, and each tap creates a brand-new charge.
• The answer can arrive another way, or not at all. The provider also confirms by calling a webhook on our server. The first version never listened, so when the first call timed out and the customer did not tap again, nothing ever told her order it was paid.
• A true-or-false flag is not a record of money. paid = true says nothing about how much, through which charge, or what happened later. When the bank’s total differs, there is nothing to match it against.

A mental model

Here are the shop counter’s two habits in full. The whole design is built from these two habits.

Payment service provider (PSP)

A company that connects a business to banks and card networks. It sends the charge to the customer’s bank, tells you the result, and later pays the collected money into your bank account, usually in one batch.

For example: The customer approves a ₹180 UPI request in their own app; the provider confirms to us within seconds and settles the day’s total the next working day.

Double-entry ledger

A record where every movement of money is written twice: once as a debit to one account and once as a credit to another, for the same amount. The sum of all debits always equals the sum of all credits.

For example: A customer pays ₹180: debit “owed to us by the PSP” ₹180, credit “sales” ₹180. The money arrives in the bank: debit “bank” ₹180, credit “owed to us by the PSP” ₹180. The receivable is back to zero.

When does the kitchen start cooking?

Two goals compete here: getting lunch out fast, and never cooking a box nobody paid for. Both protect the business.

The kitchen’s numbers are real: about 600 declined payments on 20,000 a day (at the simulator’s illustrative 3%) would be 600 boxes cooked for nobody. Cooking on success costs a few seconds on most orders and a couple of minutes on a few, which is a small price for not cooking 600 boxes a day that nobody pays for.

What to remember

1. A network call has three outcomes: it worked, it failed, or you do not know. Treat a timeout as unknown, never as failed.
2. Create a payment intent with an idempotency key before calling the provider, and send a key to the provider too. Raise the attempt number only after a definite failure.
3. Accept the provider’s answer by any road (reply, webhook, poll), record each event once in an inbox, and let a state machine that only moves forward decide the status, whatever order the events arrive in.
4. Record money in an append-only double-entry ledger. Correct mistakes with new entries, and store amounts as whole paise.
5. Reconcile every day against the settlement file. It is the only check that catches what the bank or the provider did after they answered, such as a reversed debit.

Create the payment record before you call the provider, and reconcile against the bank every day.

In the real world

Real payment systems add a great deal around this core. Providers charge fees, so a settlement line is the amount less a fee, and the ledger needs a fees account. Cards add authorisation and capture as separate steps, and disputes (chargebacks) that arrive weeks later. Refunds can be partial and can fail. Card numbers never touch your servers: the provider’s hosted page collects them, which keeps your systems out of most of the card industry’s security rules. UPI adds the customer’s approval in their own app, which is why “pending” is a normal state rather than an error.