Redesigning Parallel File Storage for AI Production: Introducing Baidu PFS L3

AI is transitioning from single‑model training to production‑level platforms that combine training, inference, agents, reinforcement learning, and simulation, all running on cloud‑native infrastructure. This shift demands a parallel file storage system that can handle dynamic, mixed workloads rather than the static, high‑performance computing (HPC) patterns of the past.

Historically, parallel file storage supported scientific computing, life sciences, and industrial simulation by providing stable, high‑bandwidth, low‑latency access for fixed‑size clusters. Early AI training resembled HPC workloads—few long‑running tasks with predictable data access—allowing traditional storage to excel.

Today, AI workloads fluctuate dramatically: training jobs may run for weeks and then release resources instantly; inference services scale with traffic; agents spin up hundreds of instances in minutes; reinforcement learning generates massive replay data; and simulation continuously creates new training samples. Consequently, storage must scale elastically with compute.

PFS L3 is built on a cloud‑native serverless architecture that decouples capacity from performance, supports online scaling, pay‑as‑you‑go pricing, and offers multiple performance tiers. It delivers up to tens of GB/s per client, total system throughput of several TB/s, IOPS in the tens of millions, and sub‑millisecond access latency.

To meet diverse data‑supply requirements, PFS L3 provides high‑bandwidth sequential reads for large model weights, high‑concurrency small‑file access for multimodal and autonomous‑driving training, rapid model loading for elastic inference, and sustained high‑throughput replay for reinforcement learning and simulation. Integrated lifecycle policies automatically keep hot data on PFS while migrating cold data to Baidu Object Storage (BOS), presenting a unified data entry point that requires no application changes.

For business collaboration, PFS L3 integrates Baidu IAM and Access Points to enforce fine‑grained, directory‑level permissions, enabling multiple teams and tenants to share the same data without duplication. Its client stack is compatible with VMs, containers, and AI training platforms, eliminating the need for OS‑specific adaptations.

"Performance exceeds PFS L2, especially for large sequential reads, noticeably accelerating model weight loading." – Baidu Wenxin

"The flexible user‑mode client deployment lets us develop and debug directly on PFS L3, boosting efficiency." – Baidu Wenku

"Stable, high‑throughput service for large AI clusters across training, inference, and reinforcement learning workloads." – Baidu Baige AI Platform

These production deployments demonstrate that AI’s storage requirements have evolved from pure performance competition to a comprehensive set of capabilities: elastic resource provisioning, continuous high‑efficiency data supply, automated lifecycle management, and cloud‑native collaboration. PFS L3 embodies this evolution, turning storage from a passive component into an active driver of AI platform efficiency.