A cache stampede in a SaaS system rarely looks dramatic in the first few minutes. Latency climbs. Error rates flicker. The database CPU graph bends upward. Somebody checks the slow query log and finds ordinary lookups that were supposed to be cached. Then traffic increases a little more, the cache misses cluster together, and the dependency that was meant to protect the database becomes the reason the database is overloaded.
A stampede happens when many requests discover the same missing or expired cache entry at once and all decide to rebuild it. One expensive computation or query becomes hundreds. The cache does not fail in an exotic way. It simply fails open at the worst possible moment, and the application does exactly what it was coded to do: fetch the source of truth.
Symptoms that point to stampedes rather than generic load
The first symptom is synchronized miss behavior. You see a sharp rise in origin load shortly after a key expires, a deploy flushes a region, or a popular tenant's working set rotates. The timing matters. Ordinary load increases smoothly with traffic. A stampede often has a cliff: fine, then suddenly not fine, clustered around a TTL boundary.
The second symptom is duplicate work with the same cache key. Application logs or tracing show many concurrent rebuilds for tenant:1842:billing-summary or org:99:nav-tree. If twenty workers are all computing the same value, you do not have a capacity problem yet. You have a coordination problem that is creating a capacity problem.
The third symptom is that the expensive path is not globally hot. It is hot for a small set of keys. Dashboard endpoints, permission snapshots, feature-flag bundles, localized navigation trees, and per-tenant configuration documents are frequent offenders. These values are read constantly, recomputed at moderate cost, and shared across many requests. That combination is stampede fuel.
A concrete example: a billing summary query takes 80 ms and is cached for sixty seconds. Under normal conditions, one request every minute rebuilds it. During a product launch email, two thousand concurrent sessions open the billing page within the same second the key expires. Without coordination, two thousand requests can attempt the rebuild. Even if the database survives, the app tier burns CPU, the connection pool saturates, and unrelated requests start waiting for connections they should have gotten immediately.
APM waterfalls help here. You will often see many parallel spans for the same downstream query starting within a few milliseconds of each other, all tagged with the same tenant or cache key. Database metrics show a burst of identical statements. Cache metrics show a miss spike followed by a fill spike. If your only view is average hit ratio for the whole cache, you can miss this. Global hit ratio can still look healthy while one key class is stampeding.
Why SaaS systems are especially exposed
Multi-tenant products concentrate popularity. A handful of large tenants, or a handful of endpoints every logged-in user hits, dominate read traffic. Caching those paths is correct. Caching them with naive get-or-compute logic is fragile.
TTL alignment makes it worse. If every tenant's configuration entry was written around deploy time with the same sixty-second TTL, many keys expire together. The system creates its own synchronized miss wave. The same thing happens after a cache flush during incident response. Operators clear Redis to fix stale data, then watch the origin melt while every process rebuilds everything at once.
Retry behavior compounds the blast. A request times out waiting for the rebuild, the client retries, and now there are more builders than original readers. Timeouts and stampedes feed each other. From the outside this looks like "the database got slow." From the inside, the database got popular for one query shape that should have been computed once.
Distributed app tiers also remove accidental protection. In a single-process system, an in-memory lock sometimes serializes rebuilds by luck. In a horizontally scaled SaaS fleet, each instance misses independently. Twelve instances with twenty threads each can create hundreds of rebuilds without any one process looking reckless.
Boring fixes that usually work
The most reliable fix is single-flight, also called request coalescing. When a miss occurs, only one caller rebuilds the value. Everyone else waits for that result or receives the previous value. In one process this can be a local mutex keyed by cache key. Across a fleet it is usually a short-lived distributed lock or a "loading" sentinel with a tight expiry. The goal is simple: one miss equals one rebuild.
Serve stale while revalidating when the data allows it. If a navigation tree that is ten seconds past TTL is still acceptable, return the stale entry and rebuild in the background. Users keep getting fast responses. The origin sees a controlled refresh instead of a synchronized crowd. This is especially useful for read-mostly reference data where brief staleness is already implicit in the TTL design.
Jitter the expiry. Instead of every key living exactly sixty seconds, expire at sixty seconds plus a random window of a few seconds. Better, set absolute expiry timestamps with jitter at write time. This breaks alignment after deploys and mass fills. It does not eliminate stampedes for a single hot key, but it prevents fleet-wide synchronized expiry across many keys.
Cap rebuild concurrency and fail closed for noncritical paths. If the lock cannot be acquired, serving a slightly stale value, a degraded payload, or a short backoff response is often better than letting every request hit Postgres. Teams resist this because degraded responses feel unfinished. During a stampede, unfinished is cheaper than cascading failure.
Precompute on write for values that change rarely and are read constantly. Tenant plan entitlements, resolved feature flags, and account display profiles often belong in this category. Update the cache when the source changes instead of discovering expiry under read traffic. You still want a TTL as a safety net. You do not want read traffic to be the primary rebuild scheduler for the hottest keys.
Keep the rebuild cheap enough that an occasional duplicate is survivable. Single-flight is the right coordination tool, but query shape still matters. If rebuilding a dashboard summary requires five sequential queries and two remote calls, even a few duplicates hurt. Materialized summaries, narrower payloads, and avoiding synchronous fan-out inside the rebuild path all reduce blast radius when coordination slips.
How to validate the fix
Prove the before and after with a focused load test against one hot key, not only with broad traffic replay. Expire the key, then send a burst of concurrent reads. Without protection you should see rebuild count climb with concurrency. With single-flight, rebuild count should stay near one while waiters attach to the in-flight computation. Measure origin query count, not just client latency. Latency can improve for coincidental reasons. Query count tells you whether duplicate work actually disappeared.
In production, alert on correlated miss spikes plus origin load for known cache key families. A global hit-ratio chart is too coarse. You want to know when billing-summary or authz-snapshot miss rates jump together with database CPU. Also watch lock wait times and rebuild durations. A single-flight system that waits on a rebuild taking five seconds can create its own latency incident even when it successfully protects the database.
Cache stampedes are not a reason to abandon caching. They are a reason to stop treating cache misses as free. In SaaS performance work, the boring controls win: coalesce rebuilds, tolerate brief staleness, stagger expiry, and keep the fallback path from inviting every request to the database at once. The systems that stay quiet under expiry are usually the ones that assumed stampedes would happen and made the failure mode dull.