Seven Key Aspects of Distributed Storage Systems: Replication, Storage Engine, Transactions, Analytics, Multi‑core, Compute, and Compilation

Motivation – Many storage concepts can be grouped into seven aspects: replication, storage engine, transactions, analytics, multi‑core, compute, and compilation.

Distributed Storage – Any storage system (object, block, file, KV, log, OLAP, OLTP) that performs partitioning and replication across multiple machines qualifies as distributed storage; partitioning provides scale, replication provides fault‑tolerance.

1. Replication – Replication is essential for availability, scalability, and performance. It introduces multi‑replica consistency problems that require consensus algorithms and replication logs. Key topics include fault detection, lease protocols, leader election, network partitions, Byzantine faults, failover, log replication (RSM), membership changes, data rebalancing, replica placement, fencing, external consistency, quorum, gossip, and distributed logging.

2. Storage Engine – The local persistent storage engine must balance CPU, memory, and device bandwidth. The "1‑3‑5" model summarizes its concerns:

1 – fsync (or equivalent) calls, their frequency and distribution across I/O.

3 – read/write/space amplification trade‑offs.

5 – the five WAL LSNs: prepare point, commit point, apply point, checkpoint, prune point.

These points enforce a total order and guide checkpointing, recovery, and log truncation. Rich data structures and algorithms, including compression and encoding, are also required for efficient memory‑disk management.

3. Transactions – Transactions provide ACID guarantees. The article discusses how transaction design reflects trade‑offs between correctness and concurrency, covering conflict detection (locking vs. timestamp ordering), visibility of committed data, isolation, and consistency. It explains pessimistic (lock‑based) and optimistic (timestamp‑based) concurrency control, the role of timestamps in ordering, and the challenges of distributed transactions (2PC/3PC) across partitions.

4. Analytics – Query processing involves parsing, semantic analysis, logical plan generation, and physical plan selection. Multiple logical and physical plans exist for a single SQL statement, differing in join order, operator implementation, predicate push‑down, and execution strategies. Execution models include tuple‑at‑a‑time (iterator), full materialization (batch), and vectorized execution (columnar, cache‑friendly loops). Columnar storage, vectorized engines, and MPP are highlighted as key technologies for modern analytical databases.

5. Multi‑core – Scaling on many cores is limited by Amdahl’s law; reducing contention is crucial. The article emphasizes lock‑free algorithms, consensus protocols, and careful task partitioning to avoid bottlenecks. It also notes the similarity between multi‑core scaling and distributed system design, where interconnect latency and memory hierarchy become dominant factors.

6. Compute – The execution engine’s roadmap is outlined, noting the lack of a baseline implementation and the need for future work.

7. Compilation – Compilation techniques permeate databases, from vectorized execution to case‑by‑case performance optimizations and heterogeneous acceleration. DSL scripts, UDFs, and compiler‑backed optimizations are mentioned as future directions.

The article concludes with a call for sharing and following the author’s channel for more architecture‑focused content.