How Off‑Heap Caching Eliminates GC Scanning for Massive Product Data

In a C‑end product service the cache size grew to gigabytes, and keeping all entries in an in‑heap Caffeine cache caused the garbage collector to scan many objects, increasing pause times.

To avoid this, the team switched to OHC (off‑heap cache), which stores data outside the JVM heap so GC does not scan the cached objects, yielding clear benefits for large‑scale caches.

Because OHC stores raw byte arrays, values must be serialized before insertion and deserialized on retrieval. The implementation uses Kryo for its speed and compactness, and a ThreadLocal provides a separate Kryo instance per thread since Kryo is not thread‑safe.

The product cache is not a single monolithic cache; it is divided by data dimensions—BASE, PRICE, SKU, BRAND, CATEGORY, IMAGE, ATTRIBUTE, DESCRIPTION—each with its own loader that handles loading, serialization, and miss handling. Each dimension can be configured with its own capacity and TTL, allowing hot base data to have larger capacity and longer TTL while low‑frequency, large description data gets tighter limits, improving overall hit rate.

OHC’s getWithLoader() method includes built‑in singleflight capability: for a given key on a machine, only one thread performs the load while others wait for the result, preventing cache stampede when hot entries expire.

In practice OHC is employed only during major promotional events as an extra local cache layer. The cache is pre‑warmed before the event, serves most requests during the event, and is bypassed afterward, reverting traffic to Redis, thus avoiding long‑term two‑level cache consistency maintenance.

Stress tests and production metrics met expectations: OHC achieved a hit rate of 92%–95%, most requests were satisfied locally, only a small fraction reached Redis, and database accesses dropped to less than 1% of total requests.