From Redis to Caffeine: Evolution of Java Caching Strategies at iQIYI

Background

This article is based on a talk at iQIYI’s tech salon about their Java caching roadmap.

Stage 1 – Data sync with Redis: data is synchronized to Redis via a message queue and Java applications read directly from the cache. Advantages: fast updates due to distributed cache. Drawbacks: heavy reliance on Redis stability; a Redis outage causes a cache avalanche and forces all requests to the database.

Stage 2 & 3 – Java Map to Guava Cache

These stages introduce an in‑process cache as a first‑level cache and Redis as a second‑level cache. Advantages: resilience to external system failures. Drawbacks: in‑process caches cannot be updated in real time and have limited memory, leading to eviction and lower hit rates.

Stage 4 – Guava Cache Refresh

Guava Cache allows setting a write‑after‑refresh interval to mitigate stale data, but it still does not provide true real‑time refresh.

Stage 5 – External Cache Asynchronous Refresh

Guava Cache is extended to use Redis as a message‑queue notification mechanism, allowing other Java applications to refresh their caches asynchronously.

Primitive Stage – Direct DB Access

When traffic is low, querying the database or reading files is the simplest solution and meets business needs.

HashMap Stage

With moderate traffic, developers may use Java’s built‑in HashMap or ConcurrentHashMap as a cache. Example code is shown in the image below.

HashMap lacks eviction, so memory can grow without bound. It is still useful in scenarios where eviction is not required, such as caching reflection metadata.

LRUHashMap Stage

To address unbounded growth, eviction algorithms like FIFO, LRU, and LFU are introduced. LRU (Least Recently Used) moves accessed entries to the tail of a linked list and evicts from the head. LFU (Least Frequently Used) tracks access frequency and evicts the least frequent entries.

Implementing an LRUMap can be done by extending LinkedHashMap and overriding removeEldestEntry.

Guava Cache

Guava Cache solves several problems of the handcrafted LRUMap:

Lock contention is reduced by segment‑level locking similar to ConcurrentHashMap.

Supports two expiration policies: expireAfterWrite and expireAfterAccess. Expiration is performed lazily during reads/writes, avoiding a global cleanup thread.

Provides automatic refresh capabilities.

Offers additional features such as weak/soft references and removal listeners for logging eviction causes.

Typical Guava Cache usage is illustrated in the following image.

Lock Competition

Guava divides the cache into segments; each segment holds at least ten entries. The number of segments is determined by concurrencyLevel (default 4). Writes acquire a lock on the relevant segment; reads acquire a lock only when the entry is absent or expired.

Expiration

Guava maintains two queues per segment: writeQueue (ordered by write time) and accessQueue (ordered by last access). When a segment exceeds its maximum size, the head of accessQueue is evicted.

Automatic Refresh

Guava performs refresh by checking a predicate during reads and invoking the configured CacheLoader when needed.

Other Features

Weak References

Both keys and values can be stored as weak or soft references; Guava tracks reclaimed entries via reference queues.

Removal Listener

A RemovalListener can be registered to receive callbacks when entries are evicted, allowing logging or custom cleanup.

RemovalCause records all eviction reasons: explicit removal, replacement, expiration, eviction, or size‑based eviction.

Caffeine – The Future

Caffeine builds on Guava’s API but implements a W‑TinyLFU policy (a hybrid of LFU and LRU) that achieves near‑optimal hit rates and superior read/write throughput.

Benchmark results show Caffeine’s hit rate approaches the theoretical optimum and its throughput dramatically exceeds Guava’s.

W‑TinyLFU

Traditional LFU records exact access frequencies, which can be costly for large key spaces. W‑TinyLFU uses a Count‑Min Sketch to approximate frequencies with limited memory, decaying counts periodically.

Read/Write Performance

Caffeine stores entries in a ConcurrentHashMap and uses three LRU queues (Eden, Probation, Protected). Write operations are queued in a RingBuffer processed by a ForkJoinPool, reducing contention. Read‑heavy workloads benefit from per‑thread RingBuffers.

Data Eviction Strategy

Caffeine’s three queues work as follows:

Eden (≈1% of capacity) holds newly added entries.

Probation holds cold entries that are candidates for eviction.

Protected (≈80% of remaining capacity) holds hot entries.

When eviction is needed, a candidate from Probation (the “attacker”) is compared with the eldest entry (the “victim”) using the frequency sketch; the entry with lower frequency is evicted, otherwise random eviction occurs.

How to Use Caffeine

The API mirrors Guava’s, making migration straightforward.

Conclusion

The article outlines iQIYI’s cache evolution from simple database queries to sophisticated multi‑level caches, highlighting the principles behind each implementation and why modern solutions like Caffeine provide better hit rates, lower latency, and higher throughput.