Mastering Distributed Systems: CAP, Redis, Zookeeper, Kafka and More

Distributed Theory

Distributed components include cache (Redis), locks (Redis or Zookeeper), services (Dubbo, Spring Cloud), coordination (Zookeeper), message queues (Kafka, RabbitMQ), sessions, transactions, and search (Elasticsearch). Mastery of one area is essential.

CAP and Consistency

CAP theorem states a distributed system can satisfy at most two of Consistency, Availability, Partition tolerance; typically CP or AP. Consistency includes strong, weak, and eventual consistency.

Distributed Transactions

Two‑phase commit (2PC) and three‑phase commit (3PC) are basic protocols. Solutions include TCC, XA, and message‑queue based compensation. TCC consists of Try, Confirm, Cancel phases with examples of resource locking and release.

Redis Overview

Redis offers in‑memory speed, rich data types (string, list, set, sorted set, hash), atomic operations, expiration, persistence (AOF, RDB), eviction policies (volatile‑LRU, allkeys‑LRU, allkeys‑random, volatile‑random, volatile‑TTL, noeviction), clustering, sentinel, and distributed locks using SET key value EX seconds NX.

Lock value can be a JSON containing thread ID and count to support re‑entrancy.

Zookeeper

Zookeeper provides coordination, distributed locks via temporary sequential nodes, leader election, and serves as a registry. It uses the Zab protocol for consistency and supports watches for event notification.

Message Queues

Message queues decouple systems, smooth spikes, and limit flow. Kafka provides high throughput, persistence, partitioning, replication, zero‑copy, and consumer groups. It uses ISR for reliability, supports various ack levels, and ensures ordering within a partition.

Other Topics

Cache pitfalls such as snowball and penetration, strategies like random expiration and Bloom filters, and high‑concurrency designs (seckill, rate limiting) are discussed.