Design and Implementation of an Outbox Redis Local Cache to Reduce Read Amplification

During high‑traffic events the dynamic feed service experiences a 2‑3× surge in feed‑stream requests, causing the outbox Redis cluster to exceed 95% CPU usage on many instances, leading to timeouts and degraded user experience.

The root cause is read amplification from the "pull" side of the outbox: each fan pulls a page of feeds from many upstream users' outbox, resulting in hundreds to thousands of times more reads than writes.

Two mitigation paths were considered: (1) increase the threshold for pushing content to users' inboxes, which would reduce outbox reads but dramatically increase inbox write pressure and storage; (2) introduce a local cache for hot upstream users (UPs) to serve recent outbox entries directly from the outbox service instance.

The chosen solution caches recent outbox entries of hot UPs in the outbox service’s local memory. Hot UPs are identified by a fan‑count threshold derived from historical traffic analysis; their latest dynamic lists are pre‑fetched from Redis and stored locally.

When an outbox instance starts, it loads the full hot‑UP list from Redis and pulls the latest entries for each. Subsequent updates are handled via a "change broadcast + asynchronous rebuild" mechanism: a change event for a cached UP triggers all instances to asynchronously refresh the local cache, avoiding a thundering‑herd of simultaneous expirations.

Consistency is ensured through two checks: (a) on‑change verification that the checksum of the local cache matches the checksum stored in Redis, and (b) a daily periodic audit comparing all cached UPs across instances with Redis; mismatches trigger another broadcast‑based rebuild.

After deployment, the local cache achieved a hit rate above 55%, reduced the outbox Redis peak load by roughly 44% during weekend peaks, and lowered the per‑instance CPU peak by 37.2%.

Future work includes extending caching to hot video and series UPs and adding a real‑time hot‑key discovery mechanism for previously uncached UPs that may become hot under specific scenarios.