Distributed email

Why did moving old receipts to cheap storage raise the bill?

Every order a partner kitchen sells now sends it a receipt, and the partner app keeps them in a receipts mailbox for each kitchen. The kitchens asked for two things: keep every receipt for seven years, and let us search them.

The first version stores each receipt, about 4 KB, as its own object in object storage (lesson 1.3). Search is a loop: list the kitchen’s receipts, read each one, keep those containing the words. With forty thousand receipts a day it works, slowly.

Older objects can live in cheaper storage classes, so two rules follow: after 30 days move a receipt to the infrequent-access class, and after a year to the archive.

A month later the storage invoice is higher than it was before the rules. Before reading on, guess how a cheaper class can cost more. The answer is in the storage price list: the cooler classes bill every object as if it were at least 128 KB, so a 4 KB receipt is billed as 32 times its size.

The same week a kitchen owner searches her mailbox for last March’s paneer orders, for her accountant. The search does not return that day. Every archived receipt has to be restored before it can be read, and the loop is waiting on hundreds of them.

Both problems have the same cause: the design treated storage as one thing with one price and one speed. Object storage is a set of tiers, each with its own prices and rules, and a search should never have to wait on the slowest tier.

Tiers, bundles and a hot index

File sync (lesson 5.5) already separated metadata from file bodies. Here the bodies also change tier as they age.

The three parts of a mailbox

• Metadata, in the database: receipt id, kitchen, date, amount, a one-line summary, and where the body lives (which object, which byte range). This data is small and always kept hot.
• Bodies, in object storage. A lifecycle rule moves them between storage tiers by age. Each tier is cheaper to keep and has stricter rules.
• The index: an inverted index (lesson 2.5) per mailbox, kept hot. Search reads the index and the metadata, never the bodies.

Storage class rules that change the design

Cooler classes cost less per GB-month but add rules: a minimum billable size per object, a minimum stay, retrieval fees, and for the coldest classes, a restore step before any read.

For example: A 4 KB receipt in an infrequent-access class is billed as 128 KB: 32 times its size. A weekly bundle of 140 receipts is 560 KB and is billed as itself.

Keep it simple, or tier it?

The single class has no new failure modes, and the first tiering attempt really was worse than doing nothing. What decides it is that the cost grows with every day of retention for seven years, while the slow path is rare: about 8.6% of opens reach the archive in the lab’s read mix, and the hot summary answers most of them without the original.

What to remember

1. Storage is a set of tiers: cheaper to keep as they get cooler, with minimum sizes, minimum stays, retrieval fees and, at the bottom, a restore step.
2. Small objects and cool tiers do not mix. Bundle them past the minimum billable size and record each item’s byte range.
3. Keep what search needs hot: metadata, a summary, and a per-mailbox inverted index. Only fetching an original may wait.
4. Partition the index by mailbox, so a search reads one kitchen’s postings and nobody else’s.

Choose where each byte is stored by how it will be read, and keep whatever search needs in the hot tier.

In the real world

Real mail systems add what this toy leaves out: retrieval fees, which can make a cool tier expensive if reads are more common than planned; compression, which suits text well; legal holds that stop deletion; per-tenant encryption keys; and a search engine that ranks results instead of returning every match. Many providers also offer a class that moves objects between tiers by watching access instead of by age, at a small fee per object.

The rule underneath does not change: the price of a byte depends on how you will read it, so design the reads first.