Skip to content
Tagged articles

activation memory

2 articles · Page 1 of 1
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Oct 28, 2026 · Artificial Intelligence

BaGuaLu-2: Runtime-Adaptive Memory Management for MoE Training on 10K+ GPU Clusters

Tsinghua University's BaGuaLu-2 system dynamically selects per-operator recomputation or offloading after MoE gating to keep activation memory within device limits, achieving 1.14× throughput over baselines on 12,288 GPUs with 32.9% average MFU, zero config changes, and open-source code.

BaGuaLuGPU clustersMoE training
0 likes · 11 min read
BaGuaLu-2: Runtime-Adaptive Memory Management for MoE Training on 10K+ GPU Clusters
Baobao Algorithm Notes
Baobao Algorithm Notes
Oct 30, 2024 · Artificial Intelligence

How Sequence Parallelism Slashes Activation Memory in Megatron Training

This article provides a detailed technical walkthrough of sequence parallelism (SP) for Megatron models, covering tensor parallelism basics, precise activation memory calculations for MLP and attention layers, the SP implementation that splits activations across GPUs, and selective activation recomputation strategies that further reduce memory while preserving training speed.

Megatronactivation memoryselective recomputation
0 likes · 20 min read
How Sequence Parallelism Slashes Activation Memory in Megatron Training