Tagged articles
model parallelism
13 articles · Page 1 of 1
This guide walks engineers through the memory, compute and bandwidth limits of training 7B‑70B models, compares data parallel (DP/DDP), tensor parallel (TP), pipeline parallel (PP), ZeRO stages and FSDP, shows how to calculate GPU memory, estimate communication overhead, configure each strategy, and avoid common pitfalls, enabling you to decide which parallelism to use on multi‑GPU or multi‑node clusters.
DeepSpeedFSDPZeRO
0 likes · 24 min read

The article dissects DeepSeek's MoE model‑parallel strategy, explaining how GPU compute and communication are overlapped through expert, pipeline, and ZeRO‑1 parallelism, and introduces DualPipe and Waved‑EP kernels that enable efficient training on large‑scale hardware.
DeepSeekGPU Communication OverlapPipeline Parallelism
0 likes · 18 min read

This article shares a detailed, first‑hand analysis of the technical challenges, framework choices, memory management, weight conversion, precision alignment, and efficiency optimizations encountered while building reinforcement‑learning pipelines for a 671‑billion‑parameter model using the verl ecosystem.
GPU memory managementMegatronRL training
0 likes · 16 min read


ArchitectMay 26, 2025 · Artificial Intelligence
This article explains the memory limits of a single GPU and systematically introduces data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and expert parallelism, describing their communication costs, advantages, drawbacks, and practical implementation details for training large AI models.
AI TrainingExpert ParallelismPipeline Parallelism
0 likes · 14 min read

The Bilibili TTV team optimized OpenSora and CogVideoX text‑to‑video models by redesigning data storage with Alluxio, parallelizing VAE encoding, applying dynamic sequence‑parallel and DeepSpeed‑Ulysses attention, adapting GPU code for NPU execution, leveraging profiling‑driven kernel fusion, FlashAttention, and expandable memory to dramatically increase training efficiency and frame throughput, while outlining future pipeline‑parallel and ZeRO‑3 scaling plans.
FlashAttentionNPUdata pipeline
0 likes · 26 min read

This article explains how the DualPipe scheduling mechanism in DeepSeek‑R1 improves GPU cluster compute‑communication efficiency by using fine‑grained pipeline stages and bidirectional data flow, comparing it with Zero Bubble pipeline parallelism and discussing the challenges of large‑scale distributed training.
DeepSeekDualPipePipeline Parallelism
0 likes · 10 min read


NewBeeNLPNov 18, 2024 · Artificial Intelligence
This article examines various techniques for compressing and accelerating the KV cache in transformer models—including MQA, GQA, MLA, sliding‑window and linear attention, flash attention, page and ring attention, as well as mixed‑precision training and ZeRO parallelism—providing code snippets, implementation details, and practical trade‑offs.
AttentionFlashAttentionKV Cache
0 likes · 17 min read

This guide walks you through the fundamentals of distributed training for large AI models, explaining data, model, and pipeline parallelism, GPU communication primitives, and advanced techniques like Megatron 3‑D parallelism and DeepSpeed ZeRO stages, with practical examples and visual illustrations to help you design efficient multi‑GPU training pipelines.
DeepSpeedGPU communicationMegatron
0 likes · 27 min read

This article explains how Alibaba Cloud's PAI platform and NVIDIA's Megatron-Core enable efficient training of sparse Mixture-of-Experts (MoE) large language models, covering algorithm basics, the Megatron-Core MoE framework, weight conversion pipelines, and performance results on Mixtral‑8x7B.
Megatron-Corelarge language modelsmixture of experts
0 likes · 18 min read

This article reviews distributed training techniques for large deep‑learning models, covering data parallelism, model parallelism (including pipeline and tensor parallelism), gradient bucketing and accumulation, 3D parallelism, and practical implementations such as Megatron‑LM and 360AI platform optimizations.
AIMegatron-LMdata parallelism
0 likes · 22 min read

This article introduces the challenges of scaling deep‑learning model training, explains the design and components of the open‑source Easy Parallel Library (EPL) that unifies data, pipeline, and operator‑split parallelism, and demonstrates its best‑practice results on large‑scale classification, BERT‑large, and massive multimodal models.
EPLParallelismZeRO
0 likes · 15 min read

The article explains how the massive compute and storage demands of today’s large language models create a “compute wall” and “storage wall,” and describes Baidu Intelligent Cloud’s four‑layer full‑stack infrastructure—combining advanced parallelism techniques, optimized GPU networking, static‑graph compilation, and cost‑model‑driven placement—to train trillion‑parameter models efficiently.
AI infrastructureGPU clusterscost-model
0 likes · 27 min read

The article reviews how deep learning models have grown deeper and wider, discusses the memory and bandwidth limits of single GPUs, and explains pipeline and sharding techniques—including GPU clusters and TPU pods—to efficiently train large‑scale models in industrial settings.
GPUPipelineTPU
0 likes · 6 min read
