PFS L3 Architecture Deep Dive: Redesigning Parallel File Storage for AI Production Workloads

AI Infrastructure Evolution Demands Storage Paradigm Shift

AI is moving from single-model training to scaled production mixing training, inference, reinforcement learning (RL), and Agent workloads. Three concurrent trends redefine parallel file storage (PFS) requirements:

Workload evolution: Multimodal, autonomous driving, and embodied AI generate >90% unstructured data (vision, point cloud, audio) with complex mixed I/O (large/small, random/sequential) and exponential growth, requiring high performance under mixed loads and cost-effective cold/hot tiering with object storage.

Architecture evolution: Post-training and RL blur training/inference boundaries. RL iterates weight updates (training) and sample rollouts (inference) on the same cluster. Checkpoints become critical-path iteration loops demanding high-concurrency writes and low-latency reads; a unified data space must allow zero-copy, zero-migration flow across preprocessing, training, fine-tuning, and inference.

Business evolution: Industry models, vertical small models, and Agent swarms bring online serving demands: billions of files with metadata latency that doesn't degrade at scale; strong isolation with per-tenant quotas, throughput QoS, and fine-grained permissions; elastic provisioning with second-level mount readiness across VMs, containers, and bare metal.

These five concrete requirements — high throughput without mixed-load degradation, horizontally scalable metadata, shared-access performance approaching local NVMe, automatic cold/hot data flow, and multi-tenant isolation with elastic provisioning — are unmet by traditional PFS born for HPC and static pre-training clusters.

PFS L3: AI-Native Parallel File System

PFS L3 is built from the ground up for AI production loads, not a port of HPC PFS. Its architecture coordinates four layers:

Client layer: Proprietary RapidFC client (kernel multi-channel, three-tier cache, zero-copy) over user-space TCP (front-end) and RDMA (data path), supporting overlay (public cloud GPU/CPU) and underlay (private RL GPU / Agent CPU) clusters.

Proxy layer: Engine proxies for request routing and aggregation.

Engine layer: BlockServer (3-replica high-performance persistent cache), Aries EC (erasure-coded low-cost persistence), MetaDB (hierarchical namespace metadata engine), and a data-flow engine bridging PFS and object storage BOS.

Five Systematic Upgrade Paths

1. Storage Engine Upgrade: Hybrid Data Engine + Unified Metadata Foundation

All-self-developed, layered architecture combining two mature engines:

Aries Linked Engine: Append-only user-space storage foundation providing online EC with arbitrary ratios (minimum 1.2x redundancy), shared with BOS and netdisk.

BlockServer 3-Replica Engine: High-performance persistent cache layer. Files split into blocks by offset, hashed uniformly across Block Replica Groups (BRGs); cluster throughput scales linearly with node count.

Write-path split at 512KB:

Large I/O (≥512KB): Direct to Aries for online EC → obtain Blob ID → write Raft log → single write, no secondary migration.

Small I/O (<512KB): Three-replica local write → quorum ACK for low latency → background compaction aggregates into Aries EC.

Result: Mixed large/small I/O performance balanced; storage redundancy cost reduced from 3x to 1.29x; single TB delivers 250 MB/s throughput.

MetaDB (internal name TafDB): Peak millions of QPS, manages 100 trillion metadata records, supports EB-scale. Single database serves three namespace types: flat (BOS), POSIX hierarchical (PFS L3, CFS), and HDFS hierarchical (AFS). Common scaling, transaction consistency, HA, and cost-efficiency pushed down; semantic differences kept upstairs. Three CCF-A top-conference papers validate the stack:

CFS: Scaling Metadata Service for Distributed File System via Pruned Scope of Critical Sections (EuroSys 2023) — POSIX namespace on MetaDB. Splits metadata across MetaDB (directory/index), FileStore (file), Renamer (slow-path rename), and client; introduces per-shard atomic primitives; removes metadata proxy. At 50 nodes, throughput 1.76–75.82× HopsFS and 1.22–4.10× InfiniFS; avg latency reduced 91.71% and 54.54%. This is the direct metadata lineage for PFS L3.

Mantle: Efficient Hierarchical Metadata Management for Cloud Object Storage Services (SOSP 2025) — HDFS namespace on MetaDB. Dual-layer IndexNode + MetaDB; full-path lookup in one RPC. vs. Tectonic/InfiniFS/LocoFS: metadata latency -6.6% to -99.1%, throughput +0.07× to +115×; interactive Spark job completion time -63.3% to -93.3%.

MetaDB: Putting File-System Structure Back into Distributed Databases (NSDI 2027) — The foundation itself. Cross-layer co-design passes directory-tree topology to sharding/execution (same-directory ops local), metadata priority to transaction grading (critical path not slowed by auxiliary updates), and deletion lifecycle to storage engine (tombstone-aware space reclamation). vs. previous general distributed DB: high-concurrency same-directory throughput +10.7×; metadata nodes reduced to <50%.

PFS L3 reuses the CFS POSIX semantic line: hierarchical namespace on MetaDB, single filesystem scales to 100B files, 4KB IOPS reaches 21K per TB.

2. Client Upgrade: RapidFC — AI-Dedicated High-Performance Client

Beyond standard NFS/SMB, RapidFC (RapidFileClient) delivers shared-storage access latency/bandwidth on par with local NVMe via three key technologies:

Kernel multi-channel: Multiple concurrent kernel channels eliminate head-of-line blocking; optimized user/kernel bidirectional communication cuts context switches and copies. Multiples of throughput vs. traditional user-space clients.

Three-tier cache: Local DRAM + local NVMe cache hot data per node; client-shared cache pool uses P2P to share model weights, datasets, checkpoints, snapshots, RAG knowledge bases across GPU nodes. Hot data hits compute-side cache; cross-cluster north-south traffic converts to intra-cluster east-west traffic, shielding backend storage from repeated storms.

Zero-copy direct access: Dual modes — standard POSIX mount (FUSE + user-space framework) for compatibility with AI frameworks, inference services, Agents, and K8s CSI; native lib SDK for performance-critical paths bypassing VFS/FUSE, building user-space zero-copy IO path aligning shared storage bandwidth with local NVMe.

Optimizations apply equally to BOS, CFS, RapidFS via same client. Aggregate: single-client throughput up to 40 GB/s, lifting per-GPU-node shared-storage ingress bandwidth and preventing IO storms on backend clusters at 10K-card scale.

3. Network Upgrade: End-to-End RDMA + User-Space TCP Deep Tuning

Target 400 Gbps fabric; currently deployed at 200 Gbps dual-NIC per node. Topology: client-side virtual networks (multiple GPU/CPU clusters) → physical Spine-Leaf backend. Three-segment tuning:

Front-end (user-space TCP): Polar TCP — RPC-oriented, polling-accelerated user-space stack with RTC (Run-To-Completion) thread model, DPDK, zero-copy, hardware offload. ~40% IOPS gain on front-end and metadata paths.

Back-end (RDMA): Congestion control, NIC firmware, queue depth, QP count tuned for storage workloads; leverages 200G multi-queue hardware. ~15% throughput uplift.

Link layer: Jumbo frames enabled. ~20% throughput gain.

Two routing optimizations:

Physical direct-connect channels: bidirectional packets traverse physical network directly from virtual-network VMs/bare metal, minimizing latency under load-balancing constraints.

Leaf-TOR hash polarization mitigation: SmartNIC and TOR have independent hash algorithms; incast or polarization creates hotspot TORs. Intelligent routing algorithms plan paths proactively, dynamically balancing source-destination TOR pairs to avoid localized hot paths.

4. Data Flow Upgrade: Transparent PFS ↔ BOS Collaboration

Bidirectional data interchange between PFS L3 and BOS lets PFS keep high-performance compute access while gaining BOS's low-cost massive persistence. Three core capabilities:

Prefetch: Sync BOS metadata or metadata+data into PFS for compute consumption.

Export: Push PFS data to BOS for backup/persistence.

Archive + Lazyload: Auto-export long-unaccessed cold data from PFS to BOS, purge local data retaining only metadata; on subsequent access, auto-fetch from BOS transparently.

Policy+task model: policies bind PFS directories to BOS buckets with prefetch/export/archive rules; tasks support auto-trigger or manual execution; preset templates for data caching, training, backup. Consistency: PFS is runtime write authority; BOS is immutable cold replica; bitmap serves as single routing truth — file reads never return stale data. Unified namespace: hot data stays in PFS, cold data auto-flows to BOS; max data-movement speed 100 GB/s.

5. Product Function Upgrade: AgenticFS for Agent-Scale Multi-Tenancy

AgenticFS manages massive independent Agent workspaces (session logs, memory, Skill scripts, RAG knowledge bases, Markdown docs) on a unified filesystem foundation. Four key mechanisms:

Sub-filesystem (FSID): Each Agent gets isolated workspace; metadata persisted in MetaDB; create/destroy are lightweight metadata ops; linear scalability to 10M+ Agents per cluster.

Directory isolation + independent Quota: Per-directory capacity and file-count quotas with real-time stats, dynamic adjustment, API-driven automation to prevent runaway storage consumption or small-file explosions.

Multi-level access points + fine-grained permissions: Flexible 1:N mapping between filesystem and sub-filesystems; per-Agent-instance read/write/traverse-deny ACLs; strict cross-space access prevention.

Protocol compatibility + elastic mount: NFS protocol, proprietary client, SDK, and K8s CSI; massive concurrent mounts with second-level elasticity matching Agent sandbox create/destroy patterns.

Production Case Study: RL Post-Training Migration

Old mode: Job submission → GPU allocation → manual 1-hour download of code/data/model → manual cross-node sync → training start. Systemic issues: GPU idle during prep; frequent missed-sync errors; any failure/restart repeats full download+sync. At ~1 failure/day, ~2 GPU-hours/day wasted per job.

New mode: PFS shared storage auto-deployed/auto-operated across resource pool; RL framework and platform fault-tolerance layer unified. Changes: (1) All nodes share single PFS directory — code, data, checkpoints, warm-start models co-located; manual sync eliminated. (2) Fault-tolerance layer restores training env from shared directory on failure — near-zero recovery cost. (3) Storage SLA 99.99%, mount guarantee 100%.

Measured gains: RL jobs default on PFS save ~30,000 GPU-hours/week (growing with RL scale). Combined with collaborative fault governance, resource-pool online rate improved from ~98% to 98.5%, foundation for 99% target.

Unified Foundation & Summary

All five layers share a single fully self-developed, serverless, elastic, EB-scale-verified foundation (MetaDB + Aries), enabling cross-product reuse and rapid evolution. PFS L3 completes the paradigm shift from high-performance storage system to AI data infrastructure, co-evolving with workloads to become the core force driving efficient collaboration of data, compute, and applications.