How Baidu’s Cloud‑Native CFS Boosts Metadata Throughput up to 75×

The test results of the paper show that in a 50‑node benchmark, Baidu’s cloud‑native file storage CFS improves throughput of all operations by 1.76‑75.82× over HopsFS and 1.22‑4.10× over InfiniFS, while reducing average latency by up to 91.71% and 54.54% respectively; under high contention and large directories, the throughput advantage expands by another order of magnitude.

Baidu Intelligent Cloud will provide an in‑depth interpretation of the paper on May 25.

Congratulations to Baidu CangHai · Storage and the University of Science and Technology of China for the paper “CFS: Scaling Metadata Service for Distributed File System via Pruned Scope of Critical Sections” being accepted to EuroSys 2023.

EuroSys (The European Conference on Computer Systems) is a top‑tier systems conference, ranked A by the China Computer Federation, alongside VLDB, FAST, and NSDI.

The 2023 EuroSys conference was held in Rome from May 8‑12, accepting 54 out of 335 submissions (acceptance rate 16.1%).

The paper introduces the core design of Baidu Intelligent Cloud’s CFS metadata system, addressing the long‑standing challenge of simultaneously achieving POSIX compatibility and high scalability (especially write scalability) in distributed file systems.

This tackles the critical question of whether a large‑scale distributed file system can scale to billions of files while maintaining high performance and stability.

The core idea is to prune the scope of critical sections to reduce lock overhead, eliminating metadata bottlenecks. The design includes:

Hierarchical, modular metadata organization that decomposes storage and processing into TafDB (directory and index), FileStore (file data), Renamer (slow‑path rename), and the client, each independently scalable.

Analysis of POSIX operation requirements leading TafDB to introduce single‑shard atomic primitives, boosting per‑shard performance and shortening request latency, thereby removing false cross‑shard conflicts.

Elimination of the traditional metadata proxy layer; the client directly provides full POSIX semantics, allowing unlimited client scaling.

The paper’s evaluation confirms that CFS’s throughput advantage grows by an order of magnitude in highly contended and large‑directory scenarios.

CFS has been running stably in production for over three years, supporting demanding big‑data, AI, container, and life‑science workloads.

About Baidu CangHai · Storage

Baidu CangHai · Storage builds a unified storage technology foundation that provides consistent capabilities for various distributed storage products, accelerating intelligent computing and unlocking data value.