Tagged articles
parallel file system
18 articles · Page 1 of 1
Baidu's PFS L3 rearchitects parallel file storage for AI production loads with a hybrid data engine (Aries + BlockServer), scalable metadata foundation (MetaDB), high-performance client (RapidFC), RDMA network tuning, transparent tiering to object storage, and AgenticFS for multi-tenant Agent workloads, proven in RL training saving 30k GPU-hours weekly.
AI storageAgenticFSAries
0 likes · 29 min read

As AI moves from isolated model training to full‑scale production with mixed workloads, Baidu's new PFS L3 offers elastic, cloud‑native parallel file storage that delivers tens of GB/s throughput, millions of IOPS, sub‑millisecond latency, and automated data lifecycle management to meet the evolving demands of modern AI platforms.
AI infrastructureBaidu CloudPFS L3
0 likes · 12 min read

The talk outlines how AI training’s growing data and compute demands create storage bottlenecks across four evolutionary stages, identifies four core problems—massive data, data‑flow, resource scheduling, and compute acceleration—and proposes hardware, software (parallel file systems, caching), and cloud‑native orchestration (Fluid, Baidu Canghai) solutions that combine object‑storage lakes with high‑performance acceleration layers to achieve near‑full GPU utilization.
AICachingStorage
0 likes · 37 min read

This article examines the comprehensive challenges AI applications face from storage to compute, traces the evolution of AI training infrastructure, analyzes key bottlenecks such as compute acceleration, resource scheduling, massive data handling and data flow, and presents Baidu Cloud's storage acceleration solutions—including parallel file systems, caching, and the Fluid scheduler—to dramatically improve AI training performance.
AI Trainingcloud nativedata lake
0 likes · 38 min read

This article explains the storage problems faced by traditional HPC, AI‑driven HPC and high‑performance data analysis, describes Baidu's internal high‑performance storage practices, and introduces the Baidu Canghai solution—including object storage BOS, parallel file system PFS, RapidFS, data‑flow mechanisms and a customer case—demonstrating how these technologies meet the demanding throughput, latency and cost requirements of modern high‑performance workloads.
AIBaiduHigh-Performance Computing
0 likes · 29 min read

This article analyzes the storage challenges of high‑performance computing—including traditional HPC, AI‑driven HPC, and high‑performance data analysis—examines Baidu’s internal practices, and presents the Canghai storage platform with its object storage, parallel file system (PFS) and RapidFS solutions that address throughput, latency, and scalability requirements.
AI TrainingHigh-Performance Computingcloud storage
0 likes · 31 min read

This article explains the storage challenges of high‑performance computing—including traditional HPC, AI‑driven HPC, and HPDA—then details Baidu’s unified storage platform, object storage BOS, and runtime solutions PFS and RapidFS, illustrating their architecture, features, and a real‑world autonomous‑driving customer case.
AI Trainingcloud storagedata lake
0 likes · 29 min read

The article introduces IBM's General Parallel File System (GPFS), describing its distributed shared‑parallel design, core components such as the management command set, kernel extensions, and daemon processes, and outlines practical deployment and performance considerations, especially in HPC environments.
GPFSHPCIBM
0 likes · 5 min read

The article provides a comprehensive overview of the Lustre parallel file system architecture, its core components, POSIX compliance, scalability, high‑performance networking, security features, data layout mechanisms, and performance considerations for large and small files, along with practical optimization tips for HPC environments.
HPCLustrePOSIX
0 likes · 17 min read

This article provides a comprehensive overview of IBM's General Parallel File System (GPFS), detailing its historical development, architectural models—including SAN, NSD, and Share‑Nothing Cluster—its operational capabilities, performance advantages, scalability, high‑availability features, and its role in large‑scale high‑performance computing environments.
GPFSHigh-Performance ComputingIBM Spectrum Scale
0 likes · 12 min read

This article explains the fundamentals of NFS, introduces the advanced pNFS architecture with its three protocols, compares storage layouts, and discusses performance benefits and real‑world deployments in high‑performance computing environments.
High-Performance ComputingNFSpNFS
0 likes · 13 min read

Burst Buffer is a storage acceleration technology that enhances I/O bandwidth and OPS for high‑performance computing by providing fast checkpoint/restart, temporary storage, and balancing SSD and parallel file system resources, with implementations from DDN, Cray, EMC, and IBM detailed for HPC designers.
Burst BufferHPCSSD
0 likes · 5 min read

BeeGFS, an open‑source parallel file system originally developed by Fraunhofer, has emerged as a flexible, high‑performance alternative to GPFS and Lustre in HPC, driven by growing demands from large‑scale analytics, AI, and cloud storage, with expanding global adoption and ecosystem partnerships.
BeeGFSGPFSHPC
0 likes · 14 min read

This article provides a comprehensive guide to optimizing Lustre, the leading open‑source parallel file system for high‑performance computing, covering network bandwidth, stripe settings, client configuration, RAID choices, small‑file handling, and practical system commands to improve aggregate I/O performance.
HPCLustrePerformance Tuning
0 likes · 8 min read

This article provides a comprehensive overview of the Lustre parallel file system, covering its architecture, key features, component roles, scalability, performance characteristics, and step‑by‑step configuration procedures for high‑performance computing environments.
ConfigurationHPCLustre
0 likes · 17 min read

This article provides a comprehensive overview of BeeGFS, covering its architecture, BeeOND on‑demand instances, quota and directory‑quota mechanisms, Buddy mirroring, supported APIs, hardware requirements, network options, and export methods via SMB/CIFS and NFS for high‑performance computing environments.
BeeGFSHigh-Performance ComputingQuota
0 likes · 11 min read

BeeGFS is a GPL‑licensed parallel file system for Linux that offers scalable storage through a modular architecture of management, metadata, and object storage servers, supports a wide range of hardware and OS platforms, and provides detailed installation, configuration, and performance‑tuning guidance including the BeeOND burst‑buffer extension.
BeeGFSHPCLinux
0 likes · 15 min read

The article provides a comprehensive overview of IBM's General Parallel File System (GPFS), detailing its physical and logical architecture, key components such as NSD and quorum mechanisms, scalability, load balancing, and fault‑tolerance features for parallel and serial applications.
GPFSHigh Availabilityparallel file system
0 likes · 11 min read
