Ultimate Cache Consistency Solution for Billion‑Scale Traffic: Canal + MQ + Local Cache

Cache consistency failure at billion‑scale traffic

Real‑world traffic shows that the simple "delete‑cache‑then‑update‑DB" pattern works for low loads (≈10 k QPS) but collapses during spikes (≈500 k–1 M QPS). Three fatal defects appear:

Cache‑deletion failure → DB contains new data while cache retains stale data permanently.

Deletion‑to‑DB update window → Reads hit stale data, write‑back propagates dirty data.

Adding a local L1 cache introduces an additional point of inconsistency with L2 Redis.

Example: during a flash‑sale, a cache‑deletion failure caused 100 000 users to see an outdated price, resulting in direct revenue loss.

Why a multi‑level cache needs a local (L1) layer

At billion‑scale traffic, Redis cost can be three times that of MySQL (e.g., MySQL HA 4C8G ≈ $7 358/year vs. Redis 4 GB master‑slave ≈ $2 448/year for equivalent storage). Shopee’s production stack uses three tiers: L1 local cache, L2 Redis, and the DB.

Request → L1 local cache (≈1 ms) → L2 Redis (≈5 ms) → DB (≈50 ms)

Typical hit‑rate and cost distribution:

L1 local cache – JVM heap/off‑heap, hit rate >80 %, cost ≈0.

L2 Redis – standalone cluster, hit rate >15 %, medium cost.

DB (MySQL) – hit rate <5 %, highest cost.

The remaining challenge is guaranteeing consistency between L1 and L2, and between Redis and the DB.

Ultimate solution architecture: Canal + MQ + Local cache

┌─────────────────────────────┐
                │   MySQL master (binlog)    │
                └───────────────┬─────────────┘
                                ▼
                ┌─────────────────────────────┐
                │          Canal              │
                │ (MySQL‑slave emulator, parses binlog) │
                └───────┬─────────────┬───────┘
                        ▼             ▼
                ┌─────────────────────────────┐
                │   RocketMQ / Kafka (MQ)    │
                │ (peak‑shaving, on‑demand subscription) │
                └───┬───────┬───────┬───────┘
                    │       │       │
                ┌───▼───┐ ┌─▼───┐ ┌─▼───┐
                │ Redis │ │Local│ │ES/HBase│
                │Cluster│ │Cache│ │Index │
                └───────┘ └─────┘ └───────┘

Component 1: Canal

Canal masquerades as a MySQL slave, requests the binlog from the master, parses it, and pushes change events to consumers.

# MySQL binlog configuration (must enable)
[mysqld]
log-bin=mysql-bin
binlog-format=ROW
server_id=1

# Grant Canal user
CREATE USER canal IDENTIFIED BY 'canal';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
FLUSH PRIVILEGES;

Three core capabilities:

Incremental subscription – only changed rows are pushed, giving millisecond latency and no DB pressure.

Data filtering – regex filters on tables/fields reduce MQ load by pushing only core data.

Multi‑source merging – data from multiple Canal instances are merged, suitable for multi‑datacenter deployments.

In production, ten Canal nodes comfortably handle tens of thousands of orders per second during Double‑11, with Kafka partitions providing data sharding.

Component 2: MQ (RocketMQ / Kafka)

Direct writes to Redis at billion‑scale would saturate it; MQ acts as a "floodgate" with three roles:

Peak‑shaving :

user flash‑sale request → MQ (ms response) → backend consumes at ~10 k/s → Redis

On‑demand subscription : only core data are published. Example with RabbitMQ topic exchange product.*.* matches all product data.

Final consistency guarantee : transactional messages + consumer offset management ensure messages are not lost, duplicated, or reordered.

RocketMQ 4.x achieves hundreds of MB/s sequential writes thanks to CommitLog ordering, PageCache, and ZeroCopy, which is orders of magnitude faster than random writes.

Component 3: Local cache update service

Without expiration, L1 can diverge from Redis; with too short TTL, DB is hit frequently. The proposed master‑slave design coordinates updates:

Coordinator (master) receives MQ messages → computes which L1 caches need updating → broadcasts update commands to workers (L1 caches)

Key design points:

Request routing – Nginx hash routing sends reads and writes for the same key to the same instance, avoiding cross‑instance inconsistency.

Short TTL fallback – L1 TTL set to 1–5 seconds; if the coordinator fails, inconsistency lasts at most a few seconds.

Version check (optional) – L1 stores a version number; reads compare it with Redis version for higher consistency (not recommended under extreme concurrency).

Local‑cache synchronization schemes compared

MQ sync (recommended) – DB → Redis → MQ → each node consumes to update L1. Reliability: ★★★★★, Performance: ★★★★, suited for billion‑scale, enterprise‑grade.

Redis Pub/Sub – replaces MQ with Redis publish/subscribe. Reliability: ★★★, Performance: ★★★★★, suited for mid‑size deployments without MQ.

Version check – L1 stores version, compares with Redis on read. Reliability: ★★★★, Performance: ★★, for low consistency requirements.

Periodic refresh – scheduled task syncs Redis data to L1. Reliability: ★★, Performance: ★★★★, auxiliary only.

Production pitfalls and mitigations

Canal single‑point failure – cache never updates. Mitigation: deploy multiple Canal instances with resume capability.

MQ consumer backlog – during peak, consumers fall behind, L1 stays stale. Mitigation: pre‑stress test and scale consumer instances.

Request routing failure – reads/writes of the same item go to different instances, L1 never consistent. Mitigation: use Nginx hash routing bound to product ID.

Too long L1 TTL – stale data persists for seconds after DB change. Mitigation: set TTL 1–3 s and rely on MQ pushes for freshness.

Full‑table push overloads Redis – Canal pushes all table changes, exhausting Redis connections. Mitigation: configure Canal table filters to push only core tables.

Choosing the right strategy

Your traffic volume
 ├─ Daily < 1 M → Cache‑Aside + delayed double‑delete
 ├─ Daily 1 M–50 M → Redisson distributed lock + Redis cache
 └─ Daily > 50 M (billion‑scale)
      ├─ Allow second‑level inconsistency?
      │   └─ Canal + RocketMQ + L1 short TTL (eventual consistency)
      └─ Require strong consistency?
          └─ Canal + MQ + local serial queue (≈30 % throughput loss)

Final takeaways

Prioritize performance: accept eventual consistency and use MQ + short TTL.

Require strong consistency: use a local serial queue, expecting ~30 % throughput reduction.

Control cost: the three‑piece combo (L1 local cache + Canal + RocketMQ) offers a balanced solution.

Core workflow: Canal captures DB changes → MQ buffers and decouples → Local cache service pushes updates to L1 → Short TTL provides a safety net, achieving final consistency.