Cache stampede explained: when one expired key overloads a database

By Unlearnt

A cache stampede happens when many requests miss the same cached value and independently fetch or rebuild it at once. The cache normally shields the database from that work. When a popular entry expires, duplicate refreshes can overwhelm the system behind it.

Worked example: 50 requests, one missing result

Imagine a product page whose recommendation list takes 200 milliseconds to calculate. At noon its cached result expires. Fifty requests arrive before the first calculation finishes, and each starts the same database work. Assume one application process, successful reads and no other traffic; these are illustrative numbers, not a benchmark.

With request coalescing, the first request starts the calculation and registers that work under the cache key. The other 49 join that in-flight result instead of starting another calculation. Once it finishes, they can use its result and later requests can use the refreshed cache.

Same requests, different handling of a cache miss.
StepIndependent refreshesCoalesced refresh
Entry expiresNo usable cached resultNo usable cached result
50 requests arrive before completion50 database calculations begin1 begins; 49 share the pending result
Refresh succeedsDuplicate results refill the keyOne result refills the key and serves the waiters

[1][2]

Stampede, stale data and thundering herd

A stale-data bug asks whether a saved value is current. A stampede asks how much repeated work happens when the value is missing or being refreshed. Fresh data does not prevent overload, and limiting refreshes does not automatically make the result correct.

Thundering herd describes a wider pattern of many workers becoming active for the same event. A cache stampede is one example. Trace the miss and refresh sequence rather than assuming every database spike is a stampede.

[3]

One refresh per process is not one refresh for the fleet

A singleflight-style group coordinates calls sharing that group. Put one group inside each of three application processes and all three may start a refresh for the same key. A shared cache does not make their separate check-then-fetch sequences atomic.

Decide whether that remaining load is acceptable. A shared refresh coordinator can reduce duplication further, but now requests depend on its availability and latency. A worker that crashes or runs beyond a lock lease also needs a defined recovery path; a lock is not a complete failure-handling policy.

[2][1]

Choose what happens while the result is missing

There is no universal best response. An old recommendation list might be acceptable briefly; an authorisation decision may need a current authoritative check. Make that choice before treating stale responses as a performance fix.

Protection and its limit.
ApproachWhat it helps withWhat still needs deciding
Coalesce requests for one keySuppress simultaneous duplicate workCoordination scope, wait time and refresh failure
Spread expiry times across keysAvoid many different keys expiring togetherOne hot key can still get simultaneous misses
Serve stale data during refreshAvoid making every reader waitMaximum staleness and who owns refresh
Bound downstream workProtect capacity during a cold cache or failureWhich requests wait, fail or receive a fallback

[3][1]

Misconception: adding randomness to the TTL fixes every stampede

Expiry jitter spreads the expiry times of different entries. It does not stop fifty requests noticing that one popular entry has expired. Probabilistic early refresh is a different technique: it gives requests a chance to refresh before expiry. Neither term should stand in for checking how duplicate work is controlled.

[1]

Check the miss path, not just the hit rate

Test what happens with a cold cache and a slow or failed refresh. A high average hit rate can hide a short burst that exhausts downstream capacity.

  • Count database refreshes per key while several requests arrive together.
  • Repeat across multiple application instances, not just one process.
  • Bound waiting and retries; ensure refresh failure does not release an unlimited wave of new attempts.
  • Record which stale results are allowed and when callers must receive an error instead.

[3]

Check your understanding

Optional: apply the idea to a new scenario.

Three application servers share a cache. Each server combines simultaneous refresh requests only within its own process. A popular key expires and requests reach all three before any refresh completes. Why can the database still receive three refreshes, and what would you weigh before coordinating refreshes across servers?

Try explaining it to the tutor →

Free to start, no account needed.

Sources

  1. Cloudflare: Sometimes I cache
  2. Go: singleflight package documentation
  3. Amazon Builders’ Library: Caching challenges and strategies