Polefs: A Cloud‑Native Distributed Cache File System for AI Training Workloads

Artificial Intelligence (AI) workloads demand massive data volumes and high‑performance GPU resources, making efficient storage and I/O crucial for cost‑effective training. Traditional single‑machine training cannot meet enterprise needs, prompting a shift toward multi‑node training platforms and cloud‑native AI infrastructure.

AI data characteristics include huge training datasets, frequent checkpointing, high I/O requirements, and metadata operations. These lead to challenges such as data redundancy, storage bottlenecks, and the need for low‑latency, high‑throughput access.

Key storage requirements derived from these characteristics are a unified storage foundation, accelerated data services, and accelerated metadata services.

The proposed solution, Polefs, is a 360‑developed, cloud‑native distributed file system that separates data and metadata. Data files are sliced and stored in object storage, while metadata resides in memory‑based clusters. The system uses a POSIX‑compatible FUSE client, enabling seamless read/write operations.

Architecture components include:

Master cluster : runs Raft for consistency, manages meta nodes and volumes, persists state in RocksDB, and provides HTTP management APIs.

Meta cluster : consists of multiple meta nodes running Multi‑Raft, stores inode and dentry information in an in‑memory B‑tree, snapshots to RocksDB, and caches metadata for high performance.

Client : implements a FUSE interface, fetches metadata from meta nodes, and accesses data from the storage cluster; supports VPC scenarios via a gateway API, and provides metadata and data caching, random writes, and large‑file uploads.

Distributed cache system : built on top of an existing distributed file system, uses object storage as the unified storage base, introduces a consistent hash ring for data sharding, and employs write‑through and read‑through caching with LRU eviction.

The write cache creates three replicas per write, uses multiple buffers, and asynchronously flushes data to object storage via a thread‑pool‑managed task system. The read cache follows a write‑then‑read order, hits the cache first, and falls back to object storage; eviction is primarily LRU‑based with capacity‑aware policies.

Failure handling includes proactive heartbeat‑based replica recovery and reactive high‑priority repair tasks.

In summary, Polefs addresses AI training’s large‑scale data, redundancy, and performance demands by providing a unified, cloud‑native storage base with accelerated data and metadata paths, improving GPU utilization and offering a scalable foundation for future AI workloads.