Design file sync
How do two devices keep the same folder without sending everything every time?
Why did a few captions cost 1.5 GB?
A home cook in Kothrud sells through the delivery service. Her menu folder holds about 400 photos, three megabytes each, and the cook app keeps it the same on her phone and her tablet.
On Sunday she prepares the new season’s menu. She sits with the tablet and adds a short caption to every photo: the dish name in Marathi and English, and whether it contains nuts.
By Sunday noon her phone has stopped loading anything. The day’s mobile data, 1.5 GB, is gone. She typed a few words on each photo, and her phone downloaded the whole folder again.
The logs explain it. A caption is stored inside the photo file, near its start. The app sees a changed file and sends the whole file: 400 photos, 1.2 GB up from the tablet, 1.2 GB down to the phone. Do the sum for one caption before reading on: 30 bytes typed, 3 MB up, 3 MB down.
On Tuesday a second fault appears. On the bus with no signal, the cook changed the price of her thali on the phone. That evening at home she corrected a spelling on the same photo on the tablet. Both synced. The price change disappeared, and customers saw the old price for two days.
Which version is the right one? The first version of the app never asks. The server keeps whichever edit reached it last and throws the other away, even when the other edit held a change the cook needed.
Pause and predict A caption changed a few bytes near the start of each 3 MB photo, and 1.2 GB moved each way. What should the app send instead?
What must never go wrong?
Start with who uses it and what sits outside it.
The most important rule in the table: once a device has saved an edit, no other edit may silently replace it.
Core entities, in two minutes
A Folder (one per cook), its Files by path, each file’s Revisions, the Blocks a revision is made of, the Devices that sync it, and a Journal of changes per folder.
The estimate that changes the design
| Quantity | Working | Result |
|---|---|---|
| Bytes stored | 2,000 cooks x 400 x 3 MB | 2.4 TB |
| New bytes a day | 2,000 x 20 x 3 MB | 120 GB |
| Upload rate, morning peak | 60 GB / 7,200 s | 8.3 MB/s |
| Commits a day | 2,000 x (20 + 30) | 100,000 |
| Commits a second, morning peak | 50,000 / 7,200 s | ≈ 7/s |
| Block-list rows, 64 KB chunks | 2,000 x 400 x (3,072 KB / 64 KB) | 38.4 million |
| Block-list size at 40 B a row | 38.4M x 40 B | 1.5 GB |
Pause and predict 2.4 TB of photo bytes that never change, and about 1.5 GB of block lists that every sync asks about. Where should each live?
The obvious version, and how it breaks
The first version is a weekend’s work. The app watches the folder. When a file changes, it uploads the whole file to the app server, which overwrites the object stored under that path and records the time. Every minute each device asks for the full listing and downloads anything newer than its copy.
Pause and predict Name one way this breaks once a cook has two devices.
A mental model
Picture a kitchen that keeps its recipes on loose cards in a binder, with a copy of the binder in a second kitchen in Nashik.
- Content addressing
- Naming a piece of data by a cryptographic hash of its bytes. Two pieces with the same name hold the same bytes, wherever they came from.
- For example: The chunk of the photo that holds the caption is stored as
blocks/9f2c…, the SHA-256 of those 61 KB. If the phone already holds a chunk with that name, it already has those bytes.
The block store is a lookup keyed by hash, and “which of these chunks do you already have?” is a set membership test: hash maps and sets in Coding Patterns.
Big blocks or small chunks?
Content-defined chunks save mobile data. Fixed blocks save rows and requests. Both costs are real.
The first path is right about captions, and the app now stores them in the database too. The gallery apps cooks use also write rotations and edits into the files, and those are outside the app’s control. The rows fit comfortably in one database. What decides it is who pays for the waste: with fixed blocks, every cook pays in mobile data on every edit.
At far larger scale the balance shifts again: many small objects cost real money in requests, so large systems pack chunks into bigger stored files and keep an index of where each chunk sits. That is an optimisation of the same idea.
What to remember
- Split the design by the shape of the data: bytes are large and immutable and go to object storage; metadata is small and answers every question.
- Cut files into chunks named by a hash of their bytes. Content addressing gives deduplication, free resumes and integrity checks.
- Content-defined chunking keeps cut points attached to the content, so an insert near the start changes one chunk instead of all of them.
- Every commit names its parent revision. A stale parent is a conflict, kept as a second file, never a silent overwrite.
- Devices sync from a cursor into a per-folder journal; notifications only say “look now”.
Send only the chunks that changed, and never let one edit silently overwrite another.
In the real world
Real sync products add much that our toy leaves out: encryption of blocks at rest, compression, sharing folders between people (which makes the folder the unit of the journal), bandwidth limits on mobile data, selective sync so a phone holds only some files, and careful handling of renames, permissions and case-insensitive file systems. Dropbox’s 2014 post describes its own choice of fixed 4 MB blocks, which favours fewer objects over smaller edits: the opposite end of the same trade-off you made here.
The parent-revision rule returns in lesson 7.2, where two people edit the same menu plan offline. There the content is text, which can be merged, so the answer will be better than a conflicted copy.