Design and Implementation of Baidu Cloud Block Storage EC System for Large‑Scale Data

The talk begins with a comparison of common data fault‑tolerance techniques, explaining why traditional RAID or simple replication is insufficient for large‑scale distributed storage and introducing erasure coding (especially Reed‑Solomon) as a cost‑effective alternative despite its higher compute and I/O overhead.

It then outlines the specific challenges of using erasure coding for block storage: frequent small‑write modifications, read‑modify‑write cycles, and I/O amplification caused by encoding/decoding, as well as the need to support both large and small I/O patterns efficiently.

Baidu's solution, built on the Aries EC system, introduces an index layer that redirects reads/writes to EC‑encoded segments and adopts an append‑only engine to avoid in‑place updates. Small writes are first cached in a three‑replica tier before being encoded in bulk, while large writes are encoded directly, reducing overall I/O amplification.

The architecture employs a two‑layer append design: a logical append engine for EC segments and a physical append engine for the underlying storage, enabling efficient space allocation and leveraging hardware‑accelerated RS encoding.

To control write amplification, Baidu proposes a cost‑benefit compaction algorithm that selects segments based on both hole ratio and data age, outperforming a simple greedy approach and achieving lower amplification especially under high space utilization.

Experimental results show that the cost‑benefit strategy keeps write amplification below 1.5× at 95% space usage, while the layered append and selective caching mechanisms provide a balanced trade‑off between cost, performance, and reliability for massive block storage workloads.