Colossal-AI: A Scalable Framework for Distributed Training of Large Models

This presentation introduces the challenges of the large‑model era, describes the Colossal‑AI architecture—including N‑dimensional parallelism, heterogeneous storage, and zero‑code experience—shows benchmark results and real‑world use cases, and answers audience questions about its integration with PyTorch and advanced parallel strategies.

AI InfrastructureBenchmarkColossal-AI

0 likes · 11 min read

Colossal-AI: A Scalable Framework for Distributed Training of Large Models

dbaplus Community

Oct 25, 2017 · Big Data

Optimizing HDFS Storage with Heterogeneous Media, Erasure Coding, and Smart Storage Management

This article explains the challenges of growing data volumes, small files, and hot‑cold data in Hadoop HDFS, then details heterogeneous storage options, erasure‑coding techniques, and the open‑source SSM (Smart Storage Management) system that automates tiered storage based on data access patterns.

Data TieringHeterogeneous StorageSmart Storage Management

0 likes · 14 min read

Optimizing HDFS Storage with Heterogeneous Media, Erasure Coding, and Smart Storage Management