CDNs and blob storage
Where should photos and files live, and how do they reach a phone far away quickly?
Why did the server stop taking orders on a Sunday?
By spring, three hundred home cooks sell through the delivery service, and the app asks each of them for photos. A photo of the dish sells more boxes than a description, so the cooks upload a lot of them.
On a Sunday evening the orders stop. The app server is alive but refuses to write anything. Its 80 GB disk is full.
Every photo came straight from a phone camera: four megabytes each, ten a day from each cook. Do the sum before reading on. 300 cooks x 10 photos x 4 MB is 12 GB a day, so an 80 GB disk lasts under a week. A bigger disk buys a few more weeks, and then the same Sunday comes back.
There is a worse problem. Since the load-balancing lesson there have been two app servers behind the load balancer. A photo uploaded through server one lives on server one’s disk. Half the customers, sent to server two, see a broken image.
And the photos that do load are slow. A regular customer who reads every ingredient says the pictures take longer to appear than his tea takes to cool. Someone in Guwahati, trying the app for a relative, gives up waiting.
The photos ended up on the app server only because that was where the code ran. The server is now storing files, sharing them between machines and delivering them to phones, and it was built for none of those jobs.
Pause and predict Two servers, photos on one server’s disk, broken images for half the customers, and a slow wait in Guwahati. Where should a photo live?
Photos on the app server’s disk
Look at what the app server was doing. It held orders and menus in the database, and photos in a folder on its own disk.
Pause and predict The accountant approves a disk ten times bigger. Name one thing that is still broken.
Putting the photos inside the database fixes the sharing problem and leaves the others. Large binary values make the database’s backups, replicas and memory carry bytes it never needs to search.
- Object storage (blob storage)
- A service that stores files of any size under a name (a key) inside a bucket, and gives them back over HTTP. You cannot edit part of an object; you replace the whole thing.
- For example:
PUT photos/k42/9f3a.jpgstores a photo.GET photos/k42/9f3a.jpgreturns it. The database keeps only the key, a few dozen bytes.
Moving the bytes to object storage solves filling and sharing. The slow photos for far-away customers need a second idea: copies kept close to them, a CDN.
A mental model
A kitchen’s supply chain has the same problem, and an old answer to it.
- CDN (content delivery network)
- Many caching servers (edges) spread across cities. A customer’s request goes to a nearby edge. If the edge has the file, it answers at once (a hit); if not, it fetches it from the origin, keeps a copy, then answers (a miss).
- For example: A Delhi customer’s photo comes from an edge in Delhi a few milliseconds away, and the origin in Mumbai is never asked.
The redesign adds two containers, and the app server is no longer in the path of any photo byte.
- Pre-signed URL
- A URL your server signs with its secret key, granting one specific action (for example, PUT this one object) until a time you choose. Whoever holds it can do that one thing and nothing else.
- For example:
PUT https://bucket…/photos/k42/9f3a.jpg?X-Amz-Expires=600&X-Amz-Signature=…, valid for 10 minutes.
Where the bytes and the milliseconds go
Start with the disk. 300 cooks x 10 photos x 4 MB = 12 GB a day. An 80 GB disk lasts 80 / 12 ≈ 6.7 days. That was the Sunday.
Now the photo a customer downloads. On a 20 Mbps phone connection, transfer time is size x 8 / bandwidth:
| Photo | Working | Transfer time |
|---|---|---|
| Straight from the camera | 4,000 KB x 8 / 20,000 kbit/s | 1,600 ms |
| Resized for a phone screen | 200 KB x 8 / 20,000 kbit/s | 80 ms |
Resizing once, at upload, saves more time than any network trick. A 4 MB photo is 20 times more bytes than the screen can show.
Then the CDN, for the 200 KB photo (the simulator’s numbers, origin work 25 ms):
| City | No CDN | Edge hit | Edge miss | Expected at 90% hits |
|---|---|---|---|---|
| Pune | 113 ms | 83 ms | 116 ms | 86 ms |
| Guwahati | 160 ms | 88 ms | 168 ms | 96 ms |
For Guwahati the expected time drops by 40%. For Pune, close to the origin, it barely moves.
The biggest saving is at the origin. At 300 photo requests a second, the origin without a CDN sends 300 x 200 KB = 60 MB/s, which is 480 Mbit/s. With a 90% hit rate it sends a tenth of that: 48 Mbit/s.
How long may an edge keep a photo?
A cook replaces a blurry photo of her thalipeeth with a sharp one. For the next hour, half the customers still see the blurry one. Edges are serving the copy they already had.
Every cached response carries a Cache-Control header that tells caches how long it stays fresh: max-age for every cache, s-maxage to override it for shared caches such as a CDN, and immutable to say the bytes behind this URL will never change.
Versioned URLs have a real cost: the menu data itself must be updated to point at the new photo, and the menu must not be cached for a year. So the menu stays on a short TTL, and only the photos, whose URLs never change meaning, get the year-long TTL.
What to remember
- Keep large files off the app server’s disk (it fills, and it is not shared between servers) and out of the database (it bloats backups and replicas). Put them in object storage and keep only the key.
- A pre-signed URL lets a phone upload straight to the bucket, so the app server never carries the bytes.
- A CDN keeps copies near customers. It helps far-away users and cuts origin bandwidth, but a miss is slower than no CDN, and it cannot fix a file that is too big.
- Cache-Control decides how long copies live. Versioned (content-hashed) URLs with a long
max-ageandimmutableavoid both staleness and purges.
Keep files in object storage, serve them through a CDN, and store only the key in the database.
In the real world
Our simulator uses illustrative round-trip times and one hit rate for every city. Real CDNs have hundreds of edges, different hit rates per region and per file, tiered caches (an origin shield), and their own fast links back to the origin. Real image pipelines also store several sizes of each photo, and modern formats such as WebP or AVIF, chosen per device.
The same split holds at any scale: the one true copy of a file lives where durable storage is cheap, and the copies people read live close to them.