How Baidu’s Unified Storage Platform Tackles AI‑Era Data Challenges

In response to the AI era’s demand for massive, high‑performance, low‑cost storage, Baidu’s Canghai Storage builds a highly reusable unified technology foundation that solves common cloud storage problems and accelerates upper‑layer system iteration.

Unified Metadata Base

The metadata base is a distributed transactional key‑value store designed for metadata scenarios, featuring a Meta‑Aware design that provides trillion‑scale metadata capacity and supports object storage (BOS) and file storage (CFS/AFS). It offers custom partitioning and co‑located mechanisms to keep related metadata on the same shard, reducing cross‑shard transaction overhead.

Meta‑Aware enables the KV system to deeply understand metadata semantics, allowing a 5‑second in‑memory MVCC with TTL, selective synchronous/asynchronous secondary indexes, and flexible engine selection (LSM‑Tree or in‑memory hash) based on access patterns.

Unified Hierarchical Namespace

The namespace evolved through three generations: an HDFS‑like single‑node design, a distributed database‑based design that sacrifices locality for scalability, and finally a single‑node‑distributed‑integrated approach that adapts seamlessly between single‑node low‑latency and distributed high‑scale modes.

Key optimizations include single‑partition transaction reduction (1PC instead of 2PC), path‑resolution acceleration via an Index shard that co‑locates directory metadata, and reduced RPCs for rename and write operations.

Unified Data Base

The data layer progressed from a master‑slave HDD‑centric architecture with 3‑replica storage, to a mixed HDD/SSD setup with offline erasure coding, and finally to a micro‑service, no‑logical‑single‑point design using online erasure coding, variable replication factors (1.5, 1.33, etc.), and PMEM acceleration.

Online EC encodes data on write, eliminating staging and reducing I/O, while the flexible replication lowers storage costs. The micro‑service decomposition improves availability, scalability, and iteration speed, supporting ZB‑scale data.