Distributed Service System Consistency: Best Practices and Patterns

The article begins by defining consistency in the context of modern internet-scale systems, distinguishing traditional strong consistency from the weaker forms required in distributed service architectures. It explains horizontal and vertical splitting of services to achieve scalability, and highlights how these splits introduce consistency challenges across multiple nodes and functional modules.

Several real‑world inconsistency cases are presented, ranging from everyday examples like mismatched housing preferences to critical financial scenarios such as transfer failures, order‑stock mismatches, synchronization timeouts, asynchronous callback delays, lost orders, and cache‑database divergence. Each case illustrates the potential impact of inconsistency on user experience and business stability.

To address these issues, the article introduces a set of practical patterns. It first discusses the classic ACID model (Atomicity, Consistency, Isolation, Durability) for strong consistency in relational databases, and contrasts it with the BASE model (Basically Available, Soft state, Eventually consistent) for scenarios where availability is prioritized.

The CAP theorem is explained, showing why distributed systems must trade off between consistency, availability, and partition tolerance. The article then details three transaction protocols: the two‑phase commit (2PC), three‑phase commit (3PC), and Alibaba’s TCC (Try‑Confirm‑Cancel), describing their workflows, advantages, and limitations such as blocking, single‑point‑of‑failure, and split‑brain problems.

Beyond protocol‑level solutions, higher‑level patterns are proposed: the Query pattern for status retrieval, the Compensation pattern for corrective actions, the Asynchronous Assurance pattern for decoupled processing, the Periodic Reconciliation pattern for detecting and fixing drift, and the Reliable Messaging pattern for ensuring durable, idempotent communication. Each pattern includes diagrams and usage recommendations for the earlier inconsistency cases.

Additional best practices for cache consistency are listed, emphasizing the use of distributed caches, full‑cache writes, and a write‑then‑read ordering to maintain weak consistency between cache and database layers.

The article concludes by summarizing the recommended approaches, noting that while strong consistency is achievable with ACID‑based databases and 2PC/3PC, many large‑scale services adopt eventual consistency through BASE, TCC, and the presented patterns to balance performance, availability, and data integrity.