Caffeine Deep Tuning: W-TinyLFU, Hit-Rate Engineering & 62%→98.7% Case Study

I. Hit Rate: The North Star of Cache Engineering

Average latency formula: avg = h * 0.5ms + (1-h) * 80ms (DB P99). At 60% hit rate → 32.2ms (cache useless); 90% → 8.5ms (passable); 99% → 1.3ms (excellent); 99.9% → 0.58ms (near limit). The last few nines deliver exponential latency reduction, but raising hit rate from 90% to 99% costs exponentially more (10× capacity? 10× TTL? consistency risk amplified). Hit-rate engineering finds the optimum in this trade-off.

Key point: Measure first, tune later. No hit-rate metric = driving blind. Micrometer + Caffeine built-in metrics (hit/miss/eviction counters) enable observability with one line of config.

II. Caffeine's Core: W-TinyLFU Eviction Algorithm

2.1 Flaws of Three Classic Policies

LRU (Least Recently Used): No defense against occasional bulk scans — one full-table scan evicts all hot keys (cache pollution).

LFU (Least Frequently Used): Historical hot keys refuse to leave — yesterday's viral product stays resident after cooling down.

FIFO: Only sees entry order, barely considers access patterns.

2.2 W-TinyLFU's Three-Region Design

Caffeine splits the queue into three regions:

Window (~1% capacity): New data enters here first, absorbs burst traffic, prevents occasional traffic from polluting the main region.

Probation (~20%): Data evicted from Window enters probation.

Protected (~80%): Data in Probation that gets accessed again promotes to Protected, enjoying longest residency.

Eviction verdict uses Count-Min Sketch : a tiny frequency-estimation array (few bits per key). It compares candidate vs victim historical access frequency; lower frequency loses.

// Count-Min Sketch intuition:
// Multiple hash functions + small array estimate access frequency
// Space O(hash_count x slots), far smaller than full counters
hash1 -> [3, 0, 7, 1, 4, ...]
hash2 -> [0, 5, 7, 2, 0, ...]
hash3 -> [1, 2, 7, 0, 3, ...]
// Query frequency = min of the three slots (collision-resistant)
// Key 7 frequency ≈ 7;
// Periodic decay prevents "historical hot keys squatting"

One-sentence summary: LRU only sees "recency", LFU only sees "frequency", W-TinyLFU sees both — Window resists burst pollution, frequency sketch + periodic decay identifies true hot keys. Official benchmarks show 10%-30% hit-rate lead over LRU and other non-Caffeine contenders.

III. Core Flow: A get() Internal Journey

Step 1: Hash to ConcurrentHashMap bucket, return value on hit.

Step 2: Record access (ReadBuffer async, read path lock-free — key to Caffeine's high throughput).

Step 3: Check expiration: expireAfterRead policy customizable.

Step 4: On miss, load via Loader/CacheLoader; write path uses WriteBuffer to async-maintain queue structures.

Step 5: Async thread (ForkJoinPool.commonPool) consumes buffers, runs eviction decisions, maintains three-region queues.

Key design: Neither read nor write path makes eviction decisions; they only drop records into buffers. Heavy lifting fully async. Trade-off: stats have slight lag — monitoring sees "near-real-time" not exact values.

IV. Five Source-Code-Level Tuning Knobs

4.1 BoundedLocalCache: True Meaning of Capacity Constraint

Caffeine.newBuilder()
// 1. By entry count: simple, but inaccurate when sizes vary
.maximumSize(10_000)
// 2. By weight: with weigher, precise memory control
.maximumWeight(64 * 1024 * 1024) // 64MB
.weigher((key, value) ->
value.length)  // weight by byte count
.build();

maximumSize counts entries; large objects cause memory explosion. Weight-based sizing tracks real memory. Caveat: weigher returning 0 evicts immediately; size() is approximate (async-maintained), don't use for precise checks.

4.2 Three Expiration Models

expireAfterWrite: Fixed TTL after write — most common, maps to "data stale at most N minutes".

expireAfterAccess: Access extends life — hot keys stay, cold data self-evicts, but may let dirty data live forever.

expireAfter(Expiry custom): Dynamic TTL per key/value — e.g., per product category freshness. Upgrades the "per cacheNames TTL" from the previous article to per-entry granularity.

// Custom Expiry: dynamic TTL by value type
Caffeine.newBuilder()
.expireAfter(
new Expiry<String, Product>() {
public long expireAfterCreate(
String k, Product v, long now) {
// Pre-sale 30s, regular 5min
return v.isPreSale()
? TimeUnit.SECONDS.toNanos(30)
: TimeUnit.MINUTES.toNanos(5);
}
public long expireAfterUpdate(
String k, Product v, long now,
long current) { return current; }
public long expireAfterRead(
String k, Product v, long now,
long current) { return current; }
})
.build();

4.3 CacheLoader: Defense Against Load Storms

LoadingCache<String, Product> cache =
Caffeine.newBuilder()
.maximumSize(10_000)
.expireAfterWrite(
Duration.ofMinutes(5))
.refreshAfterWrite(
Duration.ofMinutes(1))
.build(key -> heavyLoad(key));

// refreshAfterWrite: on expiry "return stale value first,
// background async refresh" — gentle anti-breakdown

refreshAfterWrite is the anti-breakthrough killer: expiry doesn't block; first thread gets stale value while background refreshes. Combined with expireAfterWrite: refresh failure falls back to expiry; daily penetration handled by refresh. Note: refresh executes single-threaded per key , naturally coalescing concurrent refresh requests.

4.4 RemovalListener: Eviction's "Last Will"

Caffeine.newBuilder()
.removalListener((key, value, cause) -> {
// cause: EXPLICIT/REPLACED/COLLECTED
//          / EXPIRED / SIZE
if (cause == RemovalCause.SIZE) {
metrics.increment("cache.evict.size");
}
})
.build();

Eviction cause distribution is first-hand evidence for hit-rate diagnosis: SIZE dominant = capacity too small; EXPIRED dominant = TTL too short. Listener runs on async thread by default — don't do heavy work; for synchronous awareness use evictionListener.

4.5 AsyncCache: Modern Future-Oriented API

AsyncCache<String, Product> cache =
Caffeine.newBuilder().buildAsync();

CompletableFuture<Product> future =
cache.get(key, k -> heavyLoad(k));
// Multiple threads request same key:
// Only one thread actually loads,
// others share the same Future — natural anti-breakthrough

AsyncCache.get returns the same Future — concurrent loads auto-merge, far more elegant than "lock double-check", and seamlessly integrates with the CompletableFuture world from the async article in this series.

V. Five-Step Hit-Rate Engineering

Converge tuning into five ordered actions:

Measure: Micrometer exposes hitRate/eviction gauges/counters; observe 72-hour trend.

Size: Capacity = "hot data set size" — estimate via 80/20: peak QPS × distinct keys within average TTL. Leave 30% memory headroom.

Eviction: Default maximumWeight + weigher; scan-type workloads (occasional bulk queries) handled by W-TinyLFU Window — don't hand-tweak algorithm.

TTL: Consistency requirement sets upper bound (previous article), hit-rate requirement sets lower bound; refreshAfterWrite decouples them — TTL can be long, refresh window short.

Hot-Key Governance: Sudden hit-rate cliff usually = hot key evicted or bulk invalidation. Fix: hot keys in dedicated small cache + never-expire + explicit invalidation; bulk loads stagger TTL (add random jitter, prevent avalanche).

Avalanche prevention formula: actualTTL = baseTTL × random(0.9, 1.1). One line eliminates synchronized expiry at hour boundaries.

VI. Order Service Case Study: 62% → 98.7%

Real-world transformation in four steps:

Instrument: Micrometer revealed 62% hit rate, SIZE eviction 90% — severe capacity shortage.

Re-weight: maximumSize(10_000) → maximumWeight(256MB) + byte weigher; hit rate → 81%.

Add Refresh: Product detail: refreshAfterWrite(1min) + expireAfterWrite(10min); stale value fallback, hit rate → 93%, breakthrough alerts zeroed.

Tame Bursts: Morning ops meeting triggered bulk activity-page refresh, polluting Window; hot products moved to dedicated Caffeine instance (5k entries + never-expire + explicit invalidation); overall hit rate 98.7%, DB QPS dropped 87%.

Retrospective one-liner: Low hit rate — 90% check capacity (SIZE eviction share), 10% check access pattern. Don't rush to tweak TTL.

VII. Eight Common Pitfalls

Pitfall 1: maximumSize with large objects. Entry count met but memory blown. Use weigher for byte-level control.

Pitfall 2: Treating size() as exact. Async-maintained → approximate. Wrap your own counter if needed.

Pitfall 3: CacheLoader returning null throws NPE-like eviction. Wrap optional values in Optional.

Pitfall 4: Heavy work in RemovalListener. Runs on async thread; blocks internal maintenance pool.

Pitfall 5: refreshAfterWrite configured but no Loader. Only build(LoadingCache) enables refresh semantics.

Pitfall 6: Uniform TTL for all. Synchronized expiry = scheduled avalanche. Add random jitter.

Pitfall 7: Treating Caffeine as distributed cache. In-process, per-instance independent; cross-instance consistency via broadcast invalidation (Fanout MQ, see MQ article).

Pitfall 8: Swapping Spring @Cacheable for Caffeine and done. Annotation key generation, sync semantics (sync=true) differ from direct API; breakthrough protection must be verified at @Cacheable(sync=true) or AsyncCache layer.

Three-Step Troubleshooting

Hit-rate curve shape — steady low (capacity/TTL) vs sudden drop (bulk invalidation/hot-key eviction).

Eviction cause breakdown — SIZE / EXPIRED / EXPLICIT proportions.

Sample evicted keys, check access frequency — were they truly cold or freshly hot (write storms can evict read hotspots).

VIII. Learning Path & Interview Questions

8.1 Four-Stage Roadmap

Beginner (1 week): Caffeine basic API, maximumSize/expireAfterWrite, Spring Boot integration.

Intermediate (2 weeks): weigher weight-based sizing, refreshAfterWrite anti-breakthrough, Micrometer metrics integration.

Deep (1 month): W-TinyLFU paper deep-dive (three-region + Count-Min Sketch), AsyncCache & concurrent load merging, RingBuffer async design.

Practice (ongoing): Run five-step hit-rate engineering on real services; build "eviction cause → action" runbook.

8.2 Twelve Interview Questions

Why is Caffeine fast? (lock-free read / async maintenance / W-TinyLFU)

W-TinyLFU vs LRU vs LFU? Three regions' roles?

What is Count-Min Sketch? Why take minimum?

maximumSize vs maximumWeight difference?

How to choose expireAfterWrite / expireAfterAccess / custom Expiry?

How does refreshAfterWrite prevent breakthrough? Relation to expireAfterWrite?

What happens when AsyncCache concurrent loads same key?

RemovalListener execution thread? Cause types?

Low cache hit rate — your troubleshooting steps?

How to prevent cache avalanche? (TTL jitter + multi-level + circuit breaker)

Caffeine is in-process; multi-instance consistency how?

Why is size() approximate? Design rationale?

8.3 One-Page Cheat Sheet

North Star:  Hit rate, measure first tune later
Algorithm:   W-TinyLFU = Window anti-pollution
            + frequency sketch decides life/death + periodic decay
Capacity:    Large objects use weigher by bytes
            size() is approximate
Expiry:      Freshness requirement sets TTL ceiling
            refreshAfterWrite decouples
            TTL jitter prevents avalanche
Breakthrough: AsyncCache same Future merges
            or refreshAfterWrite stale-value fallback
Diagnose:    Eviction cause SIZE=capacity
            EXPIRED=TTL short
Hot keys:    Dedicated instance + never-expire + explicit invalidation
One-liner:   Low hit rate → check capacity first, don't rush TTL

Summary: Hit-rate engineering = measure + capacity + eviction + TTL + hot-key governance, five steps in order; W-TinyLFU solves "who to evict", refreshAfterWrite solves "who loads", together they deliver high hit rate without performance loss. Local cache covered. Next up: JVM layer — Spring Boot Memory & GC Tuning: From OOM to Low-Pause Production Practice .