Seven Key Aspects of Distributed Storage Systems

Motivation: The storage domain can be divided into seven aspects—replication, storage engine, transactions, analysis, multi‑core, computation, and compilation—each addressing key challenges in building storage systems.

Distributed Storage: A system is considered distributed if it partitions and replicates data across multiple machines, providing scalability and fault tolerance.

1. Replication: Replication ensures availability, scalability, and performance by maintaining redundant copies and hot standbys. It introduces consistency challenges that are addressed by consensus algorithms and replication logs, covering fault detection, lease protocols, leader election, log replication, membership changes, data rebalancing, replica placement, fencing, external consistency, and various replication architectures.

Fault detection and lease agreements

Leader election, primary uniqueness, network partitions, split‑brain, Byzantine faults, fail‑fast/stop, failover

Log replication and state machine replication

Membership/configuration changes and scaling

Data replication, rebalancing, recovery

Replica placement and routing logic

Fencing between primary and backup

External consistency, linearizability

Replication patterns: pipeline, fan‑out, quorum, gossip

Distributed log architectures, active‑standby and active‑active setups

2. Storage Engine: The local persistent storage engine must balance CPU, memory, and device bandwidth, summarized as the 1‑3‑5 model: 1) fsync calls and their frequency, 2) read/write/space amplification trade‑offs, and 3) the five WAL LSN points (prepare, commit, apply, checkpoint, prune) that enforce a total order and guide recovery.

Data structures and algorithms: Efficient in‑memory and on‑disk structures, compression, and encoding are essential for managing data.

3. Transactions: Transactions provide ACID guarantees, with design choices affecting atomicity, consistency, isolation, and durability. Concurrency control (CC) protocols handle conflicts via lock‑based (pessimistic) or timestamp‑ordering (optimistic) methods, and distributed transactions involve commit protocols such as 2PC/3PC.

4. Analysis: Query processing transforms SQL through parsing, logical planning, and physical planning, employing optimization strategies like cost‑based models, heuristics, and adaptive execution. Execution models include tuple‑at‑a‑time, full materialization, and vectorized execution, while columnar storage and MPP architectures accelerate analytical workloads.

5. Multi‑core: Scaling on many cores is limited by Amdahl’s law; reducing contention through lock‑free algorithms, careful scheduling, and balanced task partitioning is crucial. Multi‑core scaling also resembles distributed systems, requiring efficient interconnects and memory hierarchy awareness.

6. Compute: The execution engine’s roadmap is still being defined; a baseline is needed before further details can be added.

7. Compilation: Compilation techniques enhance database performance beyond vectorization, enabling case‑by‑case optimizations, heterogeneous acceleration, and DSL‑based extensions such as UDFs.