HadaFS: A Scalable Burst Buffer File System for Exascale Supercomputers

This paper, presented at FAST 2023 by researchers from the National Supercomputing Center in Wuxi, Tsinghua University, Shandong University, and the Chinese Academy of Engineering, reports on HadaFS, a new burst‑buffer file system designed for high‑performance computing (HPC) environments.

Background: HPC workloads demand ever‑increasing I/O bandwidth, leading to the adoption of burst buffers (BB) deployed either locally on compute nodes or as shared resources. Local BBs offer scalability and performance but suffer from poor data sharing and high deployment cost, while shared BBs provide data sharing and lower cost but struggle with scalability.

Motivation: Existing BB solutions face challenges in scaling to exascale I/O concurrency, flexible consistency semantics, and dynamic data migration. HadaFS aims to unify the benefits of local and shared BBs to meet the needs of future supercomputers.

Design and Implementation: HadaFS introduces a Localized Triage Architecture (LTA) where each client connects to a single bridge server that forwards I/O requests, providing local‑BB‑like performance while enabling global sharing through a fully connected server mesh. It uses full‑path indexing for metadata, storing file metadata in key‑value stores (LMDB for local metadata, GMDB for global metadata) backed by RocksDB. The system includes a data‑management tool, Hadash, for metadata queries and data migration.

Metadata Synchronization: Three modes are offered—mode1 (asynchronous, eventual consistency), mode2 (mixed synchronous/asynchronous), and mode3 (synchronous for all operations)—allowing applications to balance consistency requirements against performance.

Performance Evaluation: Deployed on the SNS supercomputer (over 100,000 compute nodes, 600,000 HadaFS clients), HadaFS was benchmarked against BeeGFS and the native GFS (Lustre/LWFS). Metadata benchmarks (MDTest) showed HadaFS outperforming both competitors, especially at large scales. I/O bandwidth tests (IOR) demonstrated that HadaFS achieved near‑SSD theoretical limits for reads and significantly higher throughput than BeeGFS and GFS for writes. Data‑migration tests with Hadash outperformed Datawarp, achieving up to 140 GB/s for stage‑out operations and handling millions of small files efficiently.

Conclusion: HadaFS delivers a scalable, high‑performance burst‑buffer solution with flexible consistency, integrated data‑management, and proven capability to serve hundreds of applications on a supercomputer with up to 600,000 clients and an aggregate I/O bandwidth of 3.1 TB/s.