Understanding Burst Buffer Technology and Its Role in High‑Performance Computing (HPC)

Burst Buffer is a flash‑ or SSD‑based I/O acceleration layer that sits between compute nodes and parallel file systems, dramatically increasing bandwidth and file‑system OPS for high‑performance computing workloads.

At the U.S. Department of Energy's NERSC, the Cray DataWarp implementation provides Burst Buffer services for the Cori supercomputer, offering both persistent storage reservations and a cache mode for transparent performance gains.

The technology improves two key I/O scenarios: it raises the total bandwidth available to applications and boosts the number of I/O operations per second, while also enhancing checkpoint/restart reliability and providing fast temporary space for large data transfers.

Architecturally, Burst Buffer nodes are positioned between compute and storage nodes, implemented on dedicated XC40 hardware; each node contains a Xeon processor, 64 GB DDR3 memory, and two 3.2 TB NAND SSDs connected via PCIe Gen3 x8, delivering roughly 6.4 TB of capacity and 5.7 GB/s sequential throughput.

The DataWarp software stack creates mount points, manages SSDs with LVM, formats them with XFS, and presents a unified namespace to compute nodes through the DataWarp File System (DWFS), integrating tightly with the job scheduler for automatic data placement and migration.

Integration features include scheduler‑aware resource allocation, a transparent cache mode that requires no code changes, and the ability to filter or analyze data directly on Burst Buffer nodes.

Other vendors such as DDN (IME), IBM, and EMC (aBBa) offer comparable flash‑based burst buffer solutions, employing similar hardware and software concepts to accelerate I/O.

Overall, Burst Buffer technology balances storage cost and performance, extending SSD benefits beyond traditional database workloads to a wide range of HPC and data‑intensive applications.