How a Massive Cache Key Crashed Our System—and the Fixes That Saved It

Case Description

During a major promotion, a system created a huge activity whose cached data exceeded 1.5 MB. After launch, Redis call volume and latency spiked, UMP availability dropped from 100 % to 20 %, causing a cascade of failures and making the service unavailable.

Root Cause Analysis

The team used Redis as a cache and added a 5‑minute JVM local cache. However, when the activity went live the local cache was empty, causing a massive number of requests to hit Redis (cache miss). This triggered a cache‑penetration problem.

Additionally, the hot key size (≈1.5 MB) saturated the network bandwidth of a single Redis shard (default 200 Mbps ≈ 133 concurrent accesses), leading to bandwidth throttling, thread blocking, and a cache avalanche.

Solution

Implemented four measures:

Big‑key mitigation: Switched serialization from JSON to Protostuff, reducing object size from 1.5 MB to 0.5 MB.

Compression: Applied gzip compression with a threshold, shrinking a 500 KB payload to 17 KB.

Cache‑back‑origin optimization: Added a lock when the local cache misses to limit concurrent Redis fetches.

Redis monitoring & configuration: Regularly monitor network usage and adjust rate‑limit settings.

Updated cache‑fetch code:

ActivityCache present = activityLocalCache.getIfPresent(activityDetailCacheKey);
if (present != null) {
    return present;
}
ActivityCache remoteCache = getCacheFromRedis(activityDetailCacheKey);
activityLocalCache.put(activityDetailCacheKey, remoteCache);
return remoteCache;

Further refactoring introduced binary cache handling with Protostuff deserialization (code omitted for brevity).

Prevention Measures

Design stage: evaluate cache strategy and avoid large keys.

Conduct pressure testing and performance profiling before release.

Periodically optimize and upgrade the system, adopting new tools to improve stability.

Conclusion

Big‑key and hot‑key issues are common pitfalls; neglecting them can cause severe outages. Proper serialization, compression, lock‑based back‑origin, and proactive monitoring are essential to keep cache performance reliable.