MetaDB Enables Databases to Truly Understand File Systems – Accepted to NSDI’27

MetaDB, the unified metadata foundation of Baidu Cloud Storage, has been accepted to the NSDI'27 Operational Systems Track. The paper’s title, “Putting File‑System Structure Back into Distributed Databases,” captures its core claim: enabling databases to truly understand file‑system semantics.

Typical industry practice treats a distributed database as a generic black box for storing file‑system metadata, flattening the directory‑tree structure at ingestion time. This forces every request to be handled in the most expensive, generic way.

MetaDB restores the missing structural information through a cross‑layer co‑design. Compared with the previous generation of generic distributed databases, MetaDB achieves up to a 10.7× increase in throughput for high‑concurrency same‑directory operations and reduces the number of metadata machines to less than half of the original requirement.

In production, MetaDB has run stably for over five years, handling millions of QPS and managing metadata records at the 10‑trillion‑scale, supporting exabyte‑level storage. It is the first publicly described system that shows how cloud‑storage metadata workloads drive the design of a production‑grade distributed database.

The authors identify three long‑ignored drawbacks of using generic distributed databases for cloud‑storage metadata:

Databases cannot see the directory tree: over 97% of directory‑modification operations involve a single directory, yet generic databases split the data across shards, turning a local operation into a cross‑shard transaction.

Databases cannot distinguish primary from secondary metadata: auxiliary updates (e.g., parent‑directory attributes, quota statistics, secondary indexes) are treated as equal to the primary metadata, creating hotspots and slowing the critical path.

Databases are unaware of the metadata lifecycle: massive batch deletions leave behind tombstone markers that accumulate and degrade subsequent scans.

MetaDB’s solution is to pass the upper‑layer Namespace service’s knowledge of the file‑system hierarchy down to the database’s sharding, transaction, and storage layers. The directory‑tree structure guides sharding so that operations on the same directory stay local; metadata priority informs transaction tiering to keep hot paths fast; and deletion‑lifecycle patterns enable storage‑engine space reclamation without scan‑slowdown.

With this design, the database evolves from a generic black box into a foundation that truly understands file‑system semantics, providing scalable, high‑performance metadata support for massive cloud storage.

MetaDB is not an isolated result; it forms part of Baidu’s unified metadata architecture together with CFS (EuroSys'23) for POSIX semantics and Mantle (SOSP'25) for HDFS semantics. This unified base supports both flat and hierarchical namespaces, allowing performance, consistency, high‑availability, and cost‑efficiency improvements to be shared across multiple storage products such as parallel file storage (PFS L3), object storage for data lakes (BOS), and HDFS‑compatible AFS.

The unified architecture also yields “architectural compounding”: a single optimization—whether a performance boost, scalability upgrade, or reliability enhancement—benefits all product lines, turning what would be repeated engineering effort into shared, cumulative value.

MetaDB’s inclusion in NSDI'27 showcases the maturity of Baidu’s multi‑year evolution of cloud‑storage metadata systems, presenting a complete picture of the unified metadata stack to the community.