Key Aspects of Distributed Storage Systems: Replication, Engines, Transactions, Analytics, Multi‑Core, Computation, and Compilation

Motivation

The author, a senior architect, introduces the need to discuss seven fundamental aspects of storage systems: replication, storage engine, transaction, analytics, multi‑core, computation, and compilation.

Distributed Storage

Distributed storage is defined as any system that partitions and replicates data across multiple machines, regardless of the data model (object, block, file, KV, log, OLAP, OLTP).

1. Replication

Replication ensures availability, scalability, and performance, involving redundancy, hot standby, and consensus algorithms. Key topics include fault detection, lease protocols, leader election, log replication, membership changes, replica placement, external consistency, pipelines, quorum, gossip, and distributed logging.

2. Storage Engine

The storage engine focuses on persistent storage, balancing CPU, memory, and device bandwidth, summarized as the 1‑3‑5 model: 1) fsync calls and their distribution, 3) read/write/space amplification trade‑offs, and 5) the five WAL LSN points (prepare, commit, apply, checkpoint, prune) which maintain a total order.

Data Structures and Algorithms

Effective memory‑disk management relies on rich data structures, compression, and encoding algorithms to reduce size and improve performance.

3. Transaction

Transactions provide ACID guarantees; the article discusses how they expose correctness versus concurrency trade‑offs, concurrency‑control protocols (lock‑based vs. timestamp‑ordering), isolation, consistency, multi‑partition coordination, and the role of 2PC/3PC.

4. Analysis

Analytical processing involves SQL parsing, logical and physical plan generation, optimizer strategies (cost‑based, heuristic), and execution models such as tuple‑at‑a‑time, full materialization, and vectorized execution, with columnar storage and MPP being key technologies.

5. Multi‑Core

Scaling on multi‑core CPUs is limited by Amdahl’s law; reducing contention via lock‑free algorithms, careful scheduling, and considering the system as a distributed network of cores is essential.

6. Computation

The execution engine’s roadmap is outlined, emphasizing the need for a baseline before further development.

7. Compilation

Compilation techniques can enhance database performance, especially for vectorized engines, case‑by‑case optimizations, heterogeneous acceleration, and DSL‑based UDF extensions.