Cloud‑Native Storage Acceleration: Experience and Practices with CloudFS on Volcano Engine

The Volcano Engine runs most machine‑learning and data‑lake workloads on a cloud‑native Kubernetes platform, where compute, storage, and middleware are separated, driving the need for storage acceleration services.

Key challenges include lack of industry‑wide standards for storage acceleration, difficulty in selecting appropriate middleware, and governance of data flow, cost models, and protocol compatibility.

A "good" storage accelerator should provide transparent acceleration, multi‑protocol compatibility, elastic scaling, and basic data‑governance capabilities.

CloudFS, evolved from an internal HDFS implementation, offers transparent acceleration, supports native HDFS mode, and integrates with object storage and NAS backends. It provides metadata acceleration by caching and deduplicating object‑store directory trees, and data‑plane caching using block replication, ARC algorithm, lazy loading, pre‑heating via P2P, and synchronous/asynchronous write‑back.

FUSE entry points have been enhanced for higher stability, including virtio replacement, high‑availability restart, page‑cache utilization, and synchronous close support.

Practical deployments include AML platform training acceleration—where CloudFS supplies accelerated storage for GPU nodes with or without local disks—and multi‑cloud data‑lake acceleration, enabling private‑cloud caching of public‑cloud object buckets.

Performance tests show significant I/O throughput gains with caching and page‑cache enabled, and comparisons with Goofys demonstrate up to double performance after cache warm‑up.

Future plans focus on further NAS backend refinement, finer‑grained cache optimizations, and elastic scaling mechanisms for cache resources.