Key Takeaways from 'Designing Large-Scale Distributed Systems'

1. Book Overview

Book title: "Designing Large-Scale Distributed Systems". Core theme: engineering practices for building and operating large‑scale distributed systems.

2. Distributed System Basics

2.1 Definition

Distributed system = multiple independent computers + network communication + coordinated work

Features:
- Multi‑node
- Network communication
- Coordinated consistency
- Transparent to users

2.2 Distributed vs Single‑Node

Scalability: Single‑node limited, distributed theoretically unlimited.

Reliability: Single‑node has single point of failure, distributed can have redundant replicas.

Complexity: Single‑node simple, distributed complex.

Cost: Single‑node low, distributed high.

Performance: Single‑node stable, distributed may fluctuate.

3. CAP Theory

3.1 CAP Triangle

Consistency (一致性)
                     △
                    /│\
                   / │ \
                  /  │  \
                 /   │   \
Consistency◁─────●─────▷Availability
                ╱   ╲   ╱  ╲
               ╱     ╲ ╱    ╲
              ╱       ╲╱      ╲
             ╱   Partition   ╲
            ╱   Tolerance      ╲
           ╱   (分区容错)      ╲
Availability──────────────────→

3.2 Meaning of CAP

Consistency: Every read returns the latest write or an error.

Availability: Every request receives a response.

Partition Tolerance: System continues operating despite network partitions.

3.3 Choosing CAP

CP systems (strong consistency, may sacrifice availability):
- ZooKeeper
- HBase
- Redis Cluster

AP systems (high availability, may sacrifice consistency):
- Cassandra
- DynamoDB
- Eureka

CA systems: do not exist; single‑node databases are CA.

3.4 Practical Selection

Most internet scenarios: choose AP with eventual consistency.

Financial transaction scenarios: choose CP with strong consistency.

Cache scenarios: AP with TTL expiration.

4. Distributed Consistency Issues

4.1 Consistency Levels

Strong consistency: Every read sees the latest write; high latency; example: ZooKeeper.

Sequential consistency: Global order preserved; medium latency; example: Kafka.

Causal consistency: Causally related operations see order; low latency; example: Cassandra.

Eventual consistency: Reads eventually see writes; low latency; example: DynamoDB.

4.2 Distributed Transactions

Solution 1: Two‑Phase Commit (2PC)

Phase 1 – Prepare:
Coordinator → participants: ready to commit?
Participants → Coordinator: ready / fail

Phase 2 – Commit:
Coordinator → participants: commit / rollback
Participants → Coordinator: commit success / rollback success

Issues:

Synchronous blocking

Single point of failure

Data inconsistency risk

Solution 2: TCC (Try‑Confirm‑Cancel)

Try: reserve resources
Confirm: execute
Cancel: release resources

Solution 3: Saga pattern

ServiceA → ServiceB → ServiceC → ServiceD
   |        |        |        |
   ↓        ↓        ↓        ↓
Success  Success   Fail   Rollback
               ↑
               └─ Compensation action

4.3 Idempotency Design

// Non‑idempotent operation
void transfer(Account from, Account to, double amount) {
    balance -= amount; // different result if executed multiple times
}

// Idempotent operation
void transfer(Account from, Account to, double amount, String idempotentKey) {
    if (alreadyProcessed(idempotentKey)) {
        return; // duplicate execution ignored
    }
    doTransfer(from, to, amount);
    markProcessed(idempotentKey);
}

5. Load Balancing

5.1 Load‑Balancing Algorithms

Round Robin: Distribute requests sequentially; simple; does not consider load.

Random: Random selection; simple; may not be uniform.

Weighted: Distribute according to weight; controllable; requires configuration.

Least Connections: Choose node with fewest active connections; dynamic; needs computation.

Consistent Hashing: Same key maps to same node; cache‑friendly; node changes only affect neighboring nodes.

5.2 Load‑Balancing Classification

DNS load balancing: Assigns during domain resolution; tool: DNS round‑robin.

Layer‑4 load balancing: Operates at TCP/UDP layer; tool: LVS.

Layer‑7 load balancing: Operates at HTTP layer; tool: Nginx.

Client‑side load balancing: Clients select servers; tool: Ribbon.

6. Distributed Caching

6.1 Cache Strategies

Cache Aside:
  Read: cache miss → read DB → write cache
  Write: write DB → delete cache

Read Through:
  Read: cache miss → cache reads DB → return

Write Through:
  Write: write cache → write DB

Write Behind:
  Write: write cache → async write DB

6.2 Consistent Hashing

Traditional hash: hash(key) % N (remapping many keys when N changes)

Consistent hashing:
  Nodes placed on a ring; a key is stored on the first node clockwise.
  Adding/removing a node only affects its immediate neighbors, preserving most cache hits.

6.3 Redis Cluster

Redis Cluster = 16384 slots + multiple masters

Slot allocation:
- Each master owns a subset of slots.
- Clients compute slot from key and contact the corresponding node.
- During slot migration, data is copied between nodes.

Failover:
- Master failure → slave automatically promoted.
- Automatic failure detection and notification.

7. Distributed Message Queues

7.1 Role of Message Queues

Producer → Message Queue → Consumer
Features: asynchronous decoupling, peak‑shaving, reliable delivery.

7.2 Kafka Architecture

Core concepts:
- Topic: message category
- Partition: parallel unit
- Replica: high availability
- Offset: consumer position

Cluster layout (simplified):
Zookeeper/KRaft coordinates brokers.
Each broker hosts partitions with replicas (e.g., P0 R1, P1 R0, …).

7.3 Message Reliability Guarantees

At most once: May lose messages; low reliability.

At least once: May deliver duplicates; medium reliability.

Exactly once: No loss nor duplication; high reliability.

8. Distributed Coordination

8.1 ZooKeeper

Uses: configuration management, service discovery, distributed locks, leader election.

Data model: ZNode (tree node), Watch (change notification), ACL (access control).

Consistency guarantees: sequential consistency, atomicity, single view.

8.2 Distributed Lock with ZooKeeper

// ZooKeeper distributed lock
public class ZKLock {
    private ZooKeeper zk;
    private String lockPath;

    public void lock() {
        // Create an EPHEMERAL_SEQUENTIAL node
        String node = zk.create(lockPath + "/lock_", null,
                               ZooDefs.Ids.OPEN_ACL_UNSAFE,
                               CreateMode.EPHEMERAL_SEQUENTIAL);
        // Get children, smallest node gets the lock
        List<String> nodes = zk.getChildren(lockPath, true);
        Collections.sort(nodes);
        if (nodes.get(0).equals(node)) {
            // lock acquired
        } else {
            // watch previous node
            watchPrevious(node);
        }
    }
}

9. Summary

Key points: CAP theorem – choose two of three based on scenario; consistency levels have different costs; multi‑layer load balancing should be chosen wisely; cache strategies and multi‑level caching; message queues provide asynchronous decoupling and reliable transmission; coordination tools such as ZooKeeper support distributed locks and leader election.

Design principles for large‑scale distributed systems:

1. Avoid distributed transactions; prefer eventual consistency.
2. Use multi‑level caching to reduce backend pressure.
3. Employ message queues for asynchronous decoupling.
4. Design idempotent operations to ensure reliability.
5. Monitor and alert to detect problems early.

Distributed system complexity is inherent; we must manage it rather than eliminate it.