Baidu Canghai: Unified MetaDB & Aries Foundation for Object, File & Block Storage

Background and Challenges

Baidu's storage products (object storage BOS, file storage CFS, block storage) were historically built independently, leading to duplicated metadata and data-plane implementations. Object storage used distributed KV and MySQL for metadata with erasure coding (EC) for data; file and block storage relied on three-replica schemes. As data scale grew, three challenges emerged: limited metadata scalability for object and file systems, high redundancy cost of three replicas for block and file, and inefficient EC paths for small I/O and overwrite workloads.

Overall Architecture: MetaDB and Aries as Unified Foundation

The new architecture centers on two shared components: MetaDB , a distributed transactional database purpose-built for metadata, and Aries , a unified data plane providing Blob KV storage and online EC. Upper-layer products compose these building blocks via adapters: flat or hierarchical Namespace (supporting POSIX and HDFS semantics) on MetaDB, and a small-I/O/overwrite adaptation layer on Aries. For example, CFS and PFS L3 use hierarchical POSIX Namespace on MetaDB and the adaptation layer on Aries, while other products select components per their needs.

Metadata Plane: Three-Generation Evolution and MetaDB Design

First Generation: Multi-System Coexistence (2014–2017)

Each product used separate metadata systems: BOS combined MySQL and a distributed KV; file storage used single-node or subtree-partitioned schemes. These hit scaling limits as data grew.

Second Generation: Unified Distributed Architecture Exploration (2017–2019)

Inspired by the HopsFS paper (using a distributed transactional DB for file metadata), Baidu built a custom NewSQL to unify flat and hierarchical namespaces on one database. Scalability improved, but metadata performance fell short of expectations even after engineering optimizations.

Third Generation: MetaDB – Purpose-Built Distributed Transactional Database for Metadata

Recognizing a mismatch between general-purpose NewSQL design and metadata workload characteristics, Baidu developed MetaDB, co-designing sharding, transaction processing, and storage engine for metadata. MetaDB now powers BOS, CFS, and internal HDFS-compatible AFS at scale. Related papers have been accepted at SOSP, NSDI, and EuroSys.

Pain Points of Second Generation

Two partitioning schemes were evaluated: subtree partitioning (good locality for parent-child updates but poor load balance) and directory partitioning (better load distribution but splits parent directory attributes from child entries, causing cross-shard transactions and hotspot contention on parent attributes). Three specific pain points emerged:

Same-directory concurrent update conflicts: Cross-shard transactions for parent attribute + child entry updates, plus hotspot contention on the parent attribute record.

High path-resolution overhead: Serial RPC hops per path component because index is on (parent_id, name), requiring parent ID before child lookup.

Small-namespace overhead: Even when a single node could hold the entire namespace, metadata was scattered across shards, incurring distributed coordination costs.

Rebuilding Metadata Locality in Three Dimensions

1. Single-Directory Update Locality

Directory attribute separation: Parent attributes (e.g., C_attr) are split into separate records sharing the same parent_id as child entries, so they co-locate on the same shard via sharding policy, eliminating cross-shard transactions. Conflict merging: Numeric attributes (e.g., child count) use commutative delta merging; timestamp attributes use last-writer-wins. This reduces lock contention and serialization on hot parent records.

2. Path Resolution Locality with Mantle

Mantle (SOSP'25) is a hierarchical namespace service for object storage. Observation: directories are <10% of objects (billions of dirs vs tens of billions of files), so directory indexes can be centralized per namespace. Index Table (directory index) resides on a single shard; Inode Table (full metadata) remains distributed. Clients send full path to the index shard, which resolves the entire path locally in one RPC, including cycle detection for rename. Two optimizations address new bottlenecks: Stable-prefix caching (98% of renames occur at penultimate level, so cached prefixes skip repeated lookups) and Delta append for attribute updates (independent delta records avoid hotspot on the main attribute record; reads merge base + committed deltas; background compaction merges deltas).

3. Small-Scale Namespace Locality

Analysis showed >90% of filesystem instances fit within a single node's capacity and peak load. MetaDB supports a single-node mode where both Index Table and Inode Table for a namespace reside on one shard, enabling single-shard transactions and pushing full operations (path resolve + create) down to the storage node in one RPC. When capacity/load exceeds thresholds, online split and migration transitions to distributed mode without schema changes or downtime. Small scale: 1B files, ~100µs latency; large scale: 10B+ files, ms-level latency.

MetaDB Core Designs: Directory Structure, Update Semantics, Lifecycle Awareness

Directory-structure-aware sharding: Splits along directory-group boundaries to keep single-directory operations local.

Update-semantic-aware transaction processing: Deltas written to Raft log, batched in memory, asynchronously flushed; reads merge stored state with pending deltas for latest value. Auxiliary metadata (e.g., quota aggregates) updated asynchronously outside foreground transactions.

Lifecycle-aware storage engine: In-memory MVCC retains only latest committed version on flush; Raft log guarantees durability. Tracks delete-dense key ranges and triggers targeted compaction to clean tombstones, improving scan latency.

Data Plane: Three-Generation Evolution and Aries Design

First Generation: Three-Replica Architecture

Supported all products but storage cost grew with 3x redundancy.

Second Generation: Offline EC with Mola and RBS

Two-layer: Mola (three-replica staging/aggregation) → RBS (EC-encoded storage). To ease RBS Master's metadata burden (centralized fine-grained block mapping), Mola aggregated small writes into 64MB chunks before EC encoding. This reduced Master state but added staging/rewrite I/O overhead.

Third Generation: Aries – Online EC with Blob KV and Hierarchical Metadata Autonomy

Blob KV semantics: Aries exposes a Blob KV interface (addressed by Blob ID), focusing purely on data storage; directory trees remain in upper layers, avoiding directory-scaling constraints in the shared data plane. Hierarchical metadata autonomy: Master manages coarse-grained Volume topology (TB-scale logical containers composed of vlets on data nodes, e.g., 12 vlets for 8+4 EC). Data nodes locally maintain fine-grained Blob ID → physical location mappings. This shifts mapping scale with data nodes, enabling linear scalability and direct online EC writes, eliminating mandatory aggregation/staging.

Hybrid EC Adaptation for Small I/O and Overwrite

Online EC (8+4, 4KB stripe unit) amplifies small writes: a 4KB write becomes 48KB (12×4KB) vs 12KB for three replicas. Solution: Three-replica adaptation layer above Aries. Small I/O writes to replicated blocks, aggregates asynchronously, then flushes to Aries as EC. Large I/O writes directly to Aries via online EC. Overwrite semantics (required by block/file) are implemented by upper-layer logical-to-physical address mapping: appends new data, updates mapping; reads follow latest mapping. This hybrid online/offline EC architecture differs from Gen2 where all data passed through Mola due to RBS scaling limits.

Performance Optimizations: Reducing Compaction Amplification and Index Lookup Overhead

Two-Layer Engine Evolution: In-Place Update and Linked Engine

Both layers initially used append-only engines, causing read/write/CPU amplification during compaction. Upper layer (BlockServer): Evolved to File-based engine with in-place update for small I/O overwrites, reducing invalid data and subsequent compaction. Lower layer (Aries): Evolved to Linked engine with mark-delete and hole reuse . Data slices aligned to 4KB, classified by aligned length (4KB, 8KB, 16KB…). Each length class maintains a linked list of fixed-size physical blocks storing only that length. Deletion marks a hole; subsequent same-length writes reuse the hole via index, avoiding relocation of other valid records. This cuts valid-data migration, read/write amplification, CPU, and background interference.

Two-Level Index: In-Memory Offset Index + RocksDB KV Index

Write-time index records (start_offset → physical location). Reads may start at arbitrary offsets, requiring range search to find covering record. Level 1: In-memory Offset Index (bitmap: 1=record start) locates record start for a given offset instantly. Level 2: RocksDB point lookup ( Get(key)) with exact key fetches physical address. This avoids RocksDB range scans, reduces tail latency, and enables future swap to point-optimized KV engines.

Summary

Baidu Canghai storage achieves a unified technical foundation through three principles: (1) Unified foundation – MetaDB and Aries consolidate previously siloed capabilities; improvements benefit all products. (2) Scenario adaptation – Products compose different Namespace types and data-plane adapters (e.g., hybrid EC layer) atop shared components. (3) Cross-layer co-design – Upper-layer workload semantics (directory associations, update patterns, lifecycle) directly inform lower-layer mechanisms (sharding, transaction pushdown, engine design, indexing). The architecture evolves from chimney-style per-product builds to a composable, continuously evolving platform supporting object, file, and block storage at scale.