Inside a cache: how Redis works
How does one Redis process serve a hundred thousand requests a second, and what happens when its memory is full?
Why did the menus vanish at 12:29?
The menu cache from lesson 1.2 has done its job for a year. One Redis server with 2 GB of memory sits in front of PostgreSQL, and every kitchen’s lunch menu is one key in it. At 12:30 the three cities read those keys thousands of times a minute.
Then one day the order page times out again, at 12:29. For about a minute almost every menu read is a miss, and the database rebuilds two thousand menus at once. Nobody deleted anything. The menus were written at eleven with a six-hour expiry, and they should still be there.
Redis keeps its own counters. Between 12:10 and 12:25 the one called evicted_keys rose by about 300,000. Something filled the memory, and Redis threw keys out to make room.
What runs at ten past twelve is the month-end report. It reads every order from last month, through the same helper as the rest of the code: read through the cache, and on a miss, fetch from the database and store a copy. So 300,000 old orders, each read exactly once, went into Redis on their way to the accountant’s screen.
Each order was read once, yet together they pushed out the menus, which everyone reads. The cache’s memory was full, so for every new order something had to go to make room, and a setting decided which keys went.
Months ago, two lines from a blog post went into the Redis configuration: a 2 GB memory limit, and a policy called allkeys-lru. Nobody asked what the policy does with a burst of keys that will never be read again.
The first suggestion in the incident channel is that Redis is single-threaded, and that the fix is a cache with more threads and more memory. Before reading on, decide whether threads have anything to do with which keys were kept.
Pause and predict Would a cache with more threads, the same 2 GB and the same allkeys-lru policy have kept the menus?
A full cache must choose a victim
A cache in memory has a fixed size. When it is full and a new key arrives, it has three options: refuse the write, throw out an existing key, or grow until the machine runs out of memory. Redis lets you choose, with two settings.
maxmemoryis the limit for data. On a 64-bit machine the default is 0, which means no limit at all: Redis keeps allocating until the operating system swaps it to disk or kills it.maxmemory-policyis what happens at the limit. The default isnoeviction: writes that would add data fail with an error, and reads keep working. For a cache that is rarely what you want; for a store whose keys must never vanish, it is exactly right.
The configuration here said allkeys-lru: when memory is full, evict the key that was used least recently, among all keys. LRU bets that what was used recently will be used again soon.
Pause and predict At 12:10 the report pushes 300,000 orders, each read once, through a full allkeys-lru cache. Which keys does LRU throw out?
A mental model
One thread, an event loop
- Event loop
- One thread that waits for any of its many network connections to have data, handles what arrived, and goes back to waiting. It never blocks on one slow client.
- For example: Ten thousand app servers’ connections are open; the loop wakes when three of them have sent a command, runs the three commands, queues the three replies, and waits again.
Redis runs every command on one main thread, and it is still fast. The data is in memory, the structures are simple, and there are no locks, because nothing else touches the data at the same time. Redis’s own documentation puts the processing time of a typical command in the sub-microsecond range; the network round trip around it costs far more, which is why clients send many commands in one trip (pipelining, MGET).
One thread also brings a useful property: every command is atomic, meaning no other command can run partway through it. INCR on a counter cannot interleave with another INCR. The rate limiter in lesson 1.6 relied on exactly this when it ran its check-and-update as one Lua script.
The price is that one slow command makes everyone wait. KEYS * on millions of keys, reading a whole sorted set of a million members, or deleting a huge set all hold the one thread. Use SCAN to walk keys in small steps, and UNLINK to delete a big key, which frees its memory on a background thread.
Data structures beyond strings
A Redis value can be a string, a hash (a small map of fields), a list, a set, a sorted set, a stream and more. Each command works on one structure in place, so “add 5 to this dish’s score” is one command, not read, modify and write back.
- A sorted set keeps members ordered by a score. Redis implements it with a skip list (for order) plus a hash table (for finding a member), so adding or updating one member costs O(log N). The leaderboard side quest is built on one.
- Small values are packed. Small hashes, lists, sets and sorted sets use a compact encoding (a listpack since Redis 7.0) that the documentation says uses up to 10 times less memory, and convert to the full structure when they grow past a limit (128 entries for a sorted set by default).
The report, the menus and the disk
| Quantity | Working | Result |
|---|---|---|
| All kitchen menus | 2,000 x 20 KB | 40 MB |
| The report’s old orders | 300,000 x 4 KB | 1.2 GB |
| Share of a 2 GB cache the report replaced | 1.2 / 2 | 60% |
| Report inserts per second | 300,000 / 900 s | ≈ 333/s, each one an eviction |
| One core, if a command costs 10 µs end to end (an assumption) | 1 s / 10 µs | 100,000 commands/s |
The menus are 2% of the memory and nearly all of the lunch traffic. Under LRU they were also among the least recently used keys at 12:10, because the rush had not started. The report did not need to be bigger than the cache. LRU evicts the oldest keys first, so the report only had to insert more keys than the cache held keys older than the menus.
In the simulator, with 40 slots and the job on, exact LRU hits 49.6% of requests and TinyLFU admission 60.3%. That is 50.4% against 39.7% misses: about a fifth fewer trips to the database, from a change in policy alone.
What a full cache costs to keep on disk
Redis can persist its memory in two ways, and a cache must decide whether it wants either.
- RDB snapshots: Redis forks a child process, which writes the whole dataset to a compact file while the parent keeps serving. Unless configured otherwise, Redis saves after an hour if one key changed, after 5 minutes if 100 changed, and after 60 seconds if 10,000 changed. A crash loses every change since the last snapshot, which is usually a few minutes of writes.
- The append-only file (AOF): every write command is appended to a log and replayed on restart, like the write-ahead log in lesson 2.2. It is off by default (
appendonly no). With the defaultappendfsync everysec, a crash loses about the last second of writes. The settingappendfsync alwaysfsyncs before every reply, and is very slow.
Both use fork and copy-on-write. The fork itself copies the process’s page table: for 2 GB of 4 KB pages that is 2 GB / 4 KB x 8 bytes = 4 MB to copy, and the main thread pauses while it happens. Redis’s documentation lists measured forks of 9 to 13 ms per GB on physical machines and VMware, and 239 ms per GB on old Xen-based EC2 instances: from about 25 ms to about half a second for this 2 GB cache. After the fork, every memory page the parent writes during the snapshot is copied, so a busy instance can briefly need up to twice its memory.
A bigger shelf, or a better clerk?
The menus must survive the next report, and there are two ways to make sure of that.
Doubling the memory is a real fix and sometimes the right one: if the working set has grown, the cache must grow with it. Here it has not. The hot data is 40 MB; the problem is a reader that should not have been in the cache at all, and a policy that cannot tell a one-off scan from real lunch-hour popularity. LFU has a cost of its own: a new festival menu starts with a low counter, so for its first minutes it is easier to evict than a menu that has been popular all week.
What to remember
- Redis runs commands on one thread with an event loop. It is fast because everything is in memory and needs no locks, each command is atomic, and any slow command makes all the others wait.
maxmemorydefaults to no limit andmaxmemory-policytonoeviction. A cache needs both set on purpose.- LRU mistakes a scan for popularity. LFU counts how often each key is used; admission (TinyLFU) refuses newcomers that are less popular than their victim.
- Keep one-off and bulk reads out of a shared cache: a key that never enters the cache cannot push out a popular one.
- After a crash, RDB snapshots lose minutes of writes and AOF with everysec loses about a second. Both fork, and copy-on-write can briefly double memory.
LRU cannot tell a burst of one-off reads from real popularity, so keep one-off reads out of a shared cache.
In the real world
The simulator counts keys; real caches count bytes, so one large value can evict many small ones. Redis’s LFU uses a probabilistic counter and decays it by the clock, not by request count, and it samples candidates instead of scanning every key. W-TinyLFU puts a small LRU window in front of the frequency filter, so a key that is suddenly popular spends a short time in the cache, building up its count, before the frequency filter compares it with others, and Caffeine tunes the window’s size as it runs.
Recent Redis versions keep adding policies: Redis 8.6 adds allkeys-lrm, which evicts the keys least recently modified. Each policy answers the same question: when memory is full, which key should go? In lesson 3.7 the whole city opens one menu at the same moment, and one Redis instance is no longer enough.