parallel file system

Baidu Geek Talk

Nov 3, 2022 · Cloud Native

Challenges and Solutions for AI Storage Systems in Cloud‑Native Training

The talk outlines how AI training’s growing data and compute demands create storage bottlenecks across four evolutionary stages, identifies four core problems—massive data, data‑flow, resource scheduling, and compute acceleration—and proposes hardware, software (parallel file systems, caching), and cloud‑native orchestration (Fluid, Baidu Canghai) solutions that combine object‑storage lakes with high‑performance acceleration layers to achieve near‑full GPU utilization.

AICachingPerformance Optimization

0 likes · 37 min read

Challenges and Solutions for AI Storage Systems in Cloud‑Native Training

DataFunTalk

Aug 17, 2022 · Cloud Computing

High‑Performance Computing Storage Challenges and Baidu Canghai Storage Solutions

This article explains the storage problems faced by traditional HPC, AI‑driven HPC and high‑performance data analysis, describes Baidu's internal high‑performance storage practices, and introduces the Baidu Canghai solution—including object storage BOS, parallel file system PFS, RapidFS, data‑flow mechanisms and a customer case—demonstrating how these technologies meet the demanding throughput, latency and cost requirements of modern high‑performance workloads.

AIBaiduHigh Performance Computing

0 likes · 29 min read

High‑Performance Computing Storage Challenges and Baidu Canghai Storage Solutions

Architects' Tech Alliance

Dec 20, 2020 · Fundamentals

Overview of IBM GPFS (General Parallel File System) Architecture and Components

The article introduces IBM's General Parallel File System (GPFS), describing its distributed shared‑parallel design, core components such as the management command set, kernel extensions, and daemon processes, and outlines practical deployment and performance considerations, especially in HPC environments.

Distributed StorageGPFSHPC

0 likes · 5 min read

Overview of IBM GPFS (General Parallel File System) Architecture and Components

Architects' Tech Alliance

Dec 7, 2020 · Fundamentals

Overview of Lustre Parallel File System Architecture and Performance Characteristics

The article provides a comprehensive overview of the Lustre parallel file system architecture, its core components, POSIX compliance, scalability, high‑performance networking, security features, data layout mechanisms, and performance considerations for large and small files, along with practical optimization tips for HPC environments.

HPCLustrePOSIX

0 likes · 17 min read

Overview of Lustre Parallel File System Architecture and Performance Characteristics

Architects' Tech Alliance

Dec 3, 2020 · Fundamentals

IBM GPFS (Spectrum Scale) Overview: History, Architecture, Features, and High‑Performance Computing Use Cases

This article provides a comprehensive overview of IBM's General Parallel File System (GPFS), detailing its historical development, architectural models—including SAN, NSD, and Share‑Nothing Cluster—its operational capabilities, performance advantages, scalability, high‑availability features, and its role in large‑scale high‑performance computing environments.

GPFSHigh Performance ComputingIBM Spectrum Scale

0 likes · 12 min read

IBM GPFS (Spectrum Scale) Overview: History, Architecture, Features, and High‑Performance Computing Use Cases

Architects' Tech Alliance

Aug 27, 2020 · Fundamentals

Understanding Burst Buffer Technology and Its Role in High‑Performance Computing (HPC)

Burst Buffer is a storage acceleration technology that enhances I/O bandwidth and OPS for high‑performance computing by providing fast checkpoint/restart, temporary storage, and balancing SSD and parallel file system resources, with implementations from DDN, Cray, EMC, and IBM detailed for HPC designers.

Burst BufferCheckpointHPC

0 likes · 5 min read

Understanding Burst Buffer Technology and Its Role in High‑Performance Computing (HPC)

Architects' Tech Alliance

Jul 6, 2019 · Fundamentals

BeeGFS and Parallel File Systems in High‑Performance Computing: Evolution, Market Trends, and Technical Overview

BeeGFS, an open‑source parallel file system originally developed by Fraunhofer, has emerged as a flexible, high‑performance alternative to GPFS and Lustre in HPC, driven by growing demands from large‑scale analytics, AI, and cloud storage, with expanding global adoption and ecosystem partnerships.

BeeGFSGPFSHPC

0 likes · 14 min read

BeeGFS and Parallel File Systems in High‑Performance Computing: Evolution, Market Trends, and Technical Overview

Architects' Tech Alliance

Aug 7, 2018 · Operations

Lustre Performance Optimization Guide

This article provides a comprehensive guide to optimizing Lustre, the leading open‑source parallel file system for high‑performance computing, covering network bandwidth, stripe settings, client configuration, RAID choices, small‑file handling, and practical system commands to improve aggregate I/O performance.

HPCLustrePerformance Tuning

0 likes · 8 min read

Architects' Tech Alliance

Jun 25, 2018 · Fundamentals

Lustre File System: Architecture, Features, Components, and Configuration Guide

This article provides a comprehensive overview of the Lustre parallel file system, covering its architecture, key features, component roles, scalability, performance characteristics, and step‑by‑step configuration procedures for high‑performance computing environments.

HPCLustreconfiguration

0 likes · 17 min read

Lustre File System: Architecture, Features, Components, and Configuration Guide

Architects' Tech Alliance

Jul 3, 2017 · Operations

BeeGFS Features, Quotas, Mirroring, APIs, and Deployment Guidelines

This article provides a comprehensive overview of BeeGFS, covering its architecture, BeeOND on‑demand instances, quota and directory‑quota mechanisms, Buddy mirroring, supported APIs, hardware requirements, network options, and export methods via SMB/CIFS and NFS for high‑performance computing environments.

BeeGFSHigh Performance ComputingMirroring

0 likes · 11 min read

BeeGFS Features, Quotas, Mirroring, APIs, and Deployment Guidelines

Architects' Tech Alliance

Jun 24, 2017 · Operations

BeeGFS Parallel File System: Architecture, Components, Installation, and Tuning Guide

BeeGFS is a GPL‑licensed parallel file system for Linux that offers scalable storage through a modular architecture of management, metadata, and object storage servers, supports a wide range of hardware and OS platforms, and provides detailed installation, configuration, and performance‑tuning guidance including the BeeOND burst‑buffer extension.

BeeGFSHPCPerformance Tuning

0 likes · 15 min read

BeeGFS Parallel File System: Architecture, Components, Installation, and Tuning Guide

Architects' Tech Alliance

Mar 29, 2016 · Fundamentals

Overview of IBM GPFS (General Parallel File System) Architecture and Features

The article provides a comprehensive overview of IBM's General Parallel File System (GPFS), detailing its physical and logical architecture, key components such as NSD and quorum mechanisms, scalability, load balancing, and fault‑tolerance features for parallel and serial applications.

GPFSHigh Availabilityparallel file system

0 likes · 11 min read

Overview of IBM GPFS (General Parallel File System) Architecture and Features