Why Design Caches? Multi‑Level Cache Strategies, Synchronization Schemes, and Common Pitfalls

Why Design Caches?

High‑concurrency solutions are not unique to the Internet; early computer architects introduced CPU caches to bridge the speed gap between the processor and main memory, providing a small, fast storage layer for hot data.

CPU Cache Analogy

Just as a CPU uses L1, L2, and L3 caches to avoid memory bottlenecks, modern services use local JVM caches and centralized Redis caches to keep frequently accessed data close to the application.

cpu core1

cpu core2

L1d (一级数据缓存)

L1i (一级指令缓存)

L1d (一级数据缓存)

L1i (一级指令缓存)

L2

L2

L3

L3

In service‑oriented or micro‑service architectures, the database acts as persistent storage, but its throughput is far lower than that of a cache; therefore, a JVM‑level cache often outperforms a centralized Redis cache for low‑latency needs.

Give the Database Relief

Relational databases are easy to use but become a performance bottleneck as data volume grows; caching reduces read pressure and improves response time.

Distributed Cache and Multi‑Level Cache

1. Write‑through on Read

When handling a read request, first write to the local cache, then propagate to the centralized cache. Follow these principles:

Avoid copy‑and‑paste code to reduce duplication.

Keep cache logic loosely coupled from business logic for easier maintenance.

During early rollout, use feature switches sparingly; excessive switches increase system complexity. JD Logistics uses a unified configuration system called UCC.

Spring Cache can implement this pattern with a single annotation.

2. What If Writing to Cache Fails?

Maintain eventual consistency by using asynchronous messages to compensate for failures. In read‑heavy scenarios, write to the database first, then update the cache.

When the database throws an exception, re‑throw the specific exception instead of catching a generic RuntimeException, allowing targeted error handling.

3. Other Performance Considerations

Cache size should be minimal because memory is expensive; avoid storing whole objects when possible. Serialization and deserialization also add overhead.

Cache Synchronization Schemes

Different strategies balance consistency, database load, and real‑time requirements.

1. Lazy Loading

Load data into the cache on the first read; subsequent reads hit the cache. Expire entries to keep data fresh.

Pros: Simple and direct.

Cons: First request incurs a cache miss; high concurrency can overload the database.

2. Supplementary (Refresh‑Ahead)

When a cache entry nears expiration, an asynchronous worker reloads it, often using binlog‑derived messages for incremental updates.

Pros: Reduces sudden database spikes by smoothing load.

Cons: Queue buildup can cause synchronization delay and adds complexity.

3. Timed Loading

A scheduled task periodically pushes database data into a centralized cache such as Redis.

Pros: Guarantees a bounded freshness window.

Cons: Requires a reliable scheduler; adding local cache refresh increases operational cost.

PS: Monitor queue size and processing speed to avoid backlog.

Preventing Cache Penetration

Cache penetration occurs when non‑existent keys are repeatedly queried, forcing database hits. Mitigation methods:

Maintain a Set of all valid keys and check membership before querying.

Cache a placeholder (e.g., "0") for missing keys to short‑circuit DB access.

Set appropriate TTLs based on business needs, as some keys may become valid later.

Hot‑Data Cache and Eviction Policies

When only a subset of data needs to stay hot, choose an eviction strategy:

1. FIFO (First In, First Out)

Evicts the oldest entries using a simple queue.

2. LRU (Least Recently Used)

Evicts entries that have not been accessed recently; typically implemented with a hashmap + doubly linked list for O(1) operations.

3. LFU (Least Frequently Used)

Evicts entries with the lowest access count; ties are broken by recency.

Common Cache Questions

Q: Should I use a local cache or a cache cluster?

A: If the data volume is modest and updates are infrequent, a local cache is simpler; otherwise, a centralized cache scales better.

Q: How to batch‑update many cache entries?

A: Read from the database, write in bulk to the cache, and optionally use versioned keys or explicit invalidation.

Q: How to delete unknown keys periodically?

A: Instead of KEYS * , store all keys in a dedicated set and delete the set when needed.

Q: A single key holds a huge collection that cannot be evenly sharded in Redis?

A: 1) Set an expiration to tolerate occasional misses; 2) Use versioned keys (e.g., timestamp‑based) and reload based on version checks.

--- END ---