Mastering Distributed Consistency: From ACID to TCC and Beyond

Consistency Levels

Distributed systems often need to choose between strong, weak, and eventual consistency. Strong consistency guarantees that reads reflect the most recent writes but can hurt performance. Weak consistency does not promise immediate visibility of writes, while eventual consistency ensures that all replicas converge after some time, a model widely adopted in large‑scale systems.

ACID

The ACID properties—Atomicity, Consistency, Isolation, Durability—ensure reliable transaction processing in databases. A transaction is a complete logical unit of work that must either fully succeed or fully roll back, preserving data integrity.

CAP Theory

CAP states that a distributed system can simultaneously provide at most two of the following: Consistency, Availability, and Partition Tolerance. Typical trade‑offs are CA (strong consistency and availability, no partition tolerance), AP (availability and partition tolerance, weak consistency), and CP (consistency and partition tolerance, reduced availability).

BASE Theory

BASE—Basically Available, Soft state, Eventually consistent—offers an alternative to ACID for large, highly available systems. It accepts temporary inconsistency in exchange for higher availability, with the guarantee that data will become consistent over time.

Distributed Consistency Protocols

Several protocols address the coordination of distributed transactions:

Two‑Phase Commit (2PC)

2PC consists of a voting phase where the coordinator asks participants to prepare, followed by a commit phase that finalizes the transaction only if all participants agree. Drawbacks include resource blocking, a single point of failure, and potential inconsistency if the commit message is lost.

Three‑Phase Commit (3PC)

3PC adds a pre‑commit phase and timeout mechanisms to mitigate 2PC’s blocking problem. It introduces three stages—CanCommit, PreCommit, and DoCommit—but still suffers from inconsistency when network partitions occur during the PreCommit stage.

TCC (Try‑Confirm‑Cancel)

TCC moves the coordination to the service layer, allowing finer‑grained resource control. The workflow is:

Try: reserve business resources and perform validation.

Confirm: execute the actual operation idempotently.

Cancel: roll back reserved resources if any participant fails.

In a payment scenario, try reserves funds, confirm deducts them, and cancel returns them, improving concurrency compared to coarse‑grained locks in 2PC/3PC. orderClient.updateStatus(); must be split into:

orderClient.tryUpdateStatus();
orderClient.confirmUpdateStatus();
orderClient.cancelUpdateStatus();

Open‑source implementations include tcc-transaction, ByteTCC, and spring-cloud-rest-tcc.

Consistency Patterns

Practical patterns for achieving eventual consistency include:

Cache consistency: use distributed caches for high‑read workloads, accepting weak consistency between cache and database.

Query pattern: expose a status‑query API so callers can poll for operation results.

Compensation pattern: automatically retry or manually intervene to reconcile inconsistent states.

Asynchronous assurance: decouple heavy operations via message queues, processing them later while notifying callers of final outcomes.

Periodic reconciliation: run scheduled jobs that compare data across services using a globally unique identifier.

Reliable messaging: persist messages before sending, mark their status, and retry failed deliveries; ensure idempotent processing using unique indexes, Redis locks, state machines, or optimistic locking.