Fundamentals of Distributed Systems: Concepts, Replication, Consistency, and Protocols

Concepts

In a distributed system a node is typically a process on an operating system; the model treats each node as an indivisible whole. Common failures include machine crashes, network partitions, message loss, out‑of‑order delivery, and data loss, all of which must be anticipated in design.

Replication

Replication (or copy) provides redundancy for data or services. Data replicas store identical data on different nodes, while service replicas provide the same functionality without relying on local storage. Consistency models range from strong consistency (all reads see the latest write) to monotonic, session, eventual, and weak consistency, each offering different trade‑offs between correctness and performance.

Distributed System Principles

Key to scaling is how data is partitioned across nodes. Common strategies include hash‑based distribution, range‑based distribution, and chunk‑based (data‑volume) distribution. Consistent hashing improves scalability by minimizing data movement during node addition or removal, often using virtual nodes to balance load.

Replica and Data Distribution

Replica placement influences scalability and recovery speed. Storing replicas per machine is simple but limits scalability; segment‑based replication (splitting data into uniform segments) decouples replicas from physical machines, enabling faster recovery and balanced load during failures or scaling.

Replica Control Protocols

Two main families exist: centralized protocols, which rely on a single coordinator (e.g., primary‑secondary, two‑phase commit), and decentralized protocols, where all nodes are peers (e.g., Paxos). Centralized protocols are simpler but suffer from single‑point‑of‑failure and higher latency; decentralized protocols provide higher availability at the cost of complexity.

Primary‑Secondary Protocol

The primary node coordinates writes, enforces ordering, and propagates updates to secondaries. Failover requires electing a new primary, often using metadata servers to track the current primary.

Two‑Phase Commit

Involves a coordinator asking participants to prepare, collecting votes, and then committing or aborting globally. It guarantees atomicity but suffers from poor availability and performance, especially under failures.

Lease‑Based Cache

Clients receive a lease with an expiration time when caching metadata. The server guarantees no updates to the data during the lease, allowing safe caching. Lease expiration handles node or network failures gracefully.

Quorum Mechanism

Updates succeed after being written to W out of N replicas; reads succeed after reading from R replicas, with W+R > N ensuring that at least one replica overlaps, providing consistency guarantees while tolerating failures.

MVCC (Multi‑Version Concurrency Control)

Each transaction creates a new data version. Reads can select appropriate versions, enabling high concurrency without locking.

Paxos Protocol

Uses proposers, acceptors, and learners. A value is chosen when a majority of acceptors (N/2+1) accept it. Subsequent rounds can only choose the same value, providing strong consistency with fault tolerance.

CAP Theorem

States that a distributed system cannot simultaneously provide strong consistency, high availability, and partition tolerance. Different protocols make trade‑offs: lease mechanisms favor consistency and partition tolerance, quorum offers a balanced trade‑off, while two‑phase commit emphasizes consistency at the expense of availability.