Design and Evolution of Baidu Canghai Storage Unified Technology Stack

This article summarizes a presentation from the 2024 Seagate Data+ Summit about Baidu Canghai Storage’s unified technology stack, which addresses the growing demands of AI‑driven large‑scale, high‑performance, low‑cost storage.

The stack consists of three core components: a unified metadata layer, a unified hierarchical Namespace, and a unified data layer. By extracting the common metadata and data planes into reusable foundations, Baidu can build various cloud storage services (object, file, block) in a modular fashion.

Unified Metadata Layer : A distributed transactional key‑value store designed specifically for metadata, featuring a Meta‑Aware architecture that can handle trillions of metadata entries and supports BOS, CFS/AFS, and other services. The design solves the inefficiencies of generic NewSQL systems by deeply understanding metadata semantics.

Unified Namespace Layer : Built on the metadata layer, it evolved from a single‑node HDFS‑like design to a distributed architecture and finally to a single‑node‑distributed hybrid that provides both low latency for small scales and seamless horizontal scaling for large scales. Key innovations include co‑located partitioning, an Index table that reduces path‑lookup RPCs from N to 1, and optimizations for directory rename and write throughput.

Unified Data Layer : An online erasure‑coding (EC) storage system that delivers high throughput and low cost. The third‑generation design adopts a micro‑service architecture without logical single points of failure, supports ZB‑scale data, and offers both online EC and variable EC ratios (e.g., 1.5×, 1.33×) to reduce storage overhead.

The presentation also discusses the shortcomings of earlier generations (high replication cost, limited scalability, offline EC bottlenecks) and explains how the Meta‑Aware design improves partition management, transaction handling (short‑lived transactions with a 5‑second in‑memory MVCC), engine selection (LSM‑Tree vs. in‑memory hash), and SDK integration (co‑processor mechanism).

Overall, Baidu Canghai’s unified stack demonstrates a component‑based, modular approach to cloud storage that enhances performance, scalability, and maintainability while supporting diverse storage products such as BOS, CFS/AFS, and CDS.