Master Distributed Systems: CAP, Locks, Transactions, Paxos & Raft

CAP Theorem and Its Trade‑offs

The CAP theorem states that a distributed system can simultaneously guarantee at most two of three properties: Consistency, Availability, and Partition tolerance. Because network partitions are inevitable, systems must choose between strong consistency (C) and high availability (A) under partition conditions, leading to three classic models: CA (theoretical), CP (e.g., ZooKeeper), and AP (e.g., many NoSQL databases).

Distributed Lock Implementations

Three common approaches are:

MySQL lock : Create a lock table with a unique constraint; insert a row to acquire the lock and delete it to release. Simple but incurs high DB I/O and does not scale under high concurrency.

ZooKeeper lock : Use a sequential node under a designated lock directory; the client with the smallest sequence number holds the lock. Requires a ZooKeeper ensemble and is heavier than other solutions.

Redis lock : Leverage the single‑threaded nature of Redis. The basic command is SET resourceName value NX EX 5, which sets the key only if it does not exist and attaches an expiration to avoid deadlocks. Production code often uses the Redisson client, which implements the RedLock algorithm.

Distributed Transaction Patterns

Key patterns include:

Two‑Phase Commit (2PC) : Coordinator (TC) asks participants to prepare, then commits or rolls back based on unanimous readiness. Guarantees atomicity but suffers from single‑point failure and blocking.

Three‑Phase Commit (3PC) : Adds a pre‑commit phase with timeout handling to mitigate 2PC’s blocking problem.

TCC (Try‑Confirm‑Cancel) : Business‑level two‑phase commit where each operation provides explicit try, confirm, and cancel methods, reducing lock contention.

Local Message Table : Persist a message together with the business record, then asynchronously push the message to a queue; the consumer processes the message idempotently.

Message‑Based Transaction : Send a “prepare” message, execute the local transaction, then commit or rollback the message based on the transaction outcome (e.g., RocketMQ transactional messages).

Maximum‑Effort Notification : Used for low‑latency notifications such as payment callbacks; retries are limited, and a fallback query interface is provided.

Consistency Algorithms: Paxos and Raft

Paxos achieves consensus through a two‑stage process (Prepare and Accept) involving proposers, acceptors, and learners. It guarantees safety under asynchronous networks but can suffer from livelock when multiple proposers compete. Multi‑Paxos elects a leader to act as the sole proposer, improving performance.

Raft simplifies consensus by electing a leader among three roles (Leader, Follower, Candidate). The leader handles log replication and sends periodic heartbeats; followers start a new election if heartbeats are missed. Raft’s election and log‑replication steps are described in detail, making it easier to implement than Paxos.

Idempotency Techniques

Common strategies to ensure that repeated requests have the same effect include:

Pre‑check with SELECT before INSERT using a unique request identifier.

Enforce unique indexes and handle duplicate‑key exceptions.

Apply pessimistic locks ( SELECT ... FOR UPDATE) or optimistic locks with version/timestamp columns.

Maintain a deduplication table that records processed request IDs.

Model business state machines so that only valid state transitions are allowed.

Use distributed locks (e.g., Redis + Redisson) for critical sections.

Issue a one‑time token to the client and verify its uniqueness on the server.

SELECT * FROM user WHERE id=123 FOR UPDATE;

UPDATE user SET amount=amount+100, version=version+1 WHERE id=123 AND version=1;

Distributed Rate Limiting

Three typical algorithms:

Counter : Simple fixed‑window counting; suffers from burst spikes.

Leaky Bucket : Models a bucket that drains at a constant rate; excess requests are queued or dropped.

Token Bucket : Tokens are added at a steady rate; each request consumes a token. Implementations include Guava’s RateLimiter for single‑instance services.

References

《从Paxos到Zookeeper 分布式一致性原理与实践》

《分布式理论(一) - CAP定理》 https://juejin.cn/post/6844903621490901006

《分布式理论(二) - BASE理论》 https://juejin.cn/post/6844903621495095304

《分布式理论(三) - 2PC协议》 https://juejin.cn/post/6844903621495095309

《从分布式事务解决到Seata使用，一梭子给你整明白了》 https://juejin.cn/post/6944882663148748807

《高并发下如何保证接口的幂等性?》 https://juejin.cn/post/6944559294939398158

《诸葛亮 VS 庞统，拿下 Paxos 共识算法》 http://www.passjava.cn/#/03.Distributed/05.诸葛VS庞统，拿下Paxos