Unlocking 10K‑GPU LLM Training: Inside MegaScale’s 55% MFU Breakthrough

Background and Motivation

Training large language models (LLMs) requires enormous compute resources; leading AI players now operate clusters with tens of thousands of GPUs. When scaling beyond ten thousand GPUs, two critical challenges arise: maintaining high training efficiency (measured by Model FLOPs Utilization, MFU) and ensuring stable, fault‑tolerant execution.

Key Challenges in 10K‑GPU Clusters

Efficiency: MFU reflects the ratio of actual throughput to theoretical maximum. Distributed LLM training demands extensive inter‑GPU communication, operator optimization, data preprocessing, and careful GPU memory management.

Stability: Failures and delays are common at this scale; a single lagging GPU can stall the entire job, making rapid fault detection and recovery essential.

Megascale System Overview

MegaScale, deployed in ByteDance’s data center, addresses these challenges through a tightly coupled algorithm‑system co‑design.

Algorithm Optimizations

Parallel Transformer Blocks: Replace the serial transformer formulation with a parallel version, allowing attention and MLP blocks to run concurrently without degrading model quality.

Sliding‑Window Attention (SWA): A sparse attention mechanism that limits each token’s context to a fixed‑size window, reducing computation while preserving long‑range information.

LAMB Optimizer: Enables batch sizes up to 64K for BERT‑style training without loss of accuracy, overcoming the usual batch‑size ceiling.

3D Parallel Communication Overlap

MegaScale employs tensor, pipeline, and data parallelism. Communication operations (all‑gather and reduce‑scatter) are overlapped with computation by pre‑fetching all‑gather at the start of each iteration and decoupling send/receive in the pipeline’s 1F1B schedule. Inspired by PyTorch FSDP, this reduces hidden communication phases.

Efficient Operators

Adoption of FlashAttention‑2 for attention kernels, improving thread‑block and warp scheduling.

Fused LayerNorm and GeLU kernels to cut kernel launch overhead and improve memory access patterns.

Data‑Pipeline Optimizations

Asynchronous Preprocessing: Data preprocessing runs off the critical path, overlapping with gradient synchronization.

Eliminating Redundant Loaders: A single shared‑memory loader per machine serves all GPU workers in the same tensor‑parallel group, removing duplicate disk reads and boosting transfer efficiency.

Collective Communication Group Initialization

Standard torch.distributed initialization becomes a bottleneck beyond a few thousand GPUs. MegaScale replaces the blocking TCPStore with a non‑blocking Redis backend and redesigns the barrier sequence to minimize global synchronizations. Initialization time drops from 1,047 s (2,048 GPUs) to under 5 s, and below 30 s for >10k GPUs.

Network Performance Tuning

Topology built on Broadcom Tomahawk 4 switches (25.6 Tbps total bandwidth, 64 × 400 Gbps ports) in a CLOS‑style three‑tier layout.

ECMP hash collision reduction by separating uplink/downlink ports and using split 400 G links.

Hybrid congestion control combining Swift and DCQCN, with precise RTT measurement and ECN feedback to mitigate head‑of‑line blocking.

Adjusted NCCL retransmission timers and enabled adap_retrans on NICs for faster recovery from link jitter.

Fault Tolerance and Monitoring

A custom Kubernetes interface launches a dedicated pod per node. Each executor runs a watchdog that sends heartbeat messages; missing heartbeats trigger automatic checkpoint restoration and pod eviction/replacement. The system also provides millisecond‑level monitoring of key metrics and a fast checkpoint‑restore pipeline.

Results and Conclusions

When training a 175 B parameter LLM on 12,288 GPUs, MegaScale achieved 55.2 % MFU—1.34× higher than Megatron‑LM—demonstrating that algorithm‑system co‑design can substantially improve both efficiency and stability at extreme scale. The authors emphasize the importance of fault‑tolerant design and comprehensive monitoring for future LLM training research.

Source: SDNLAB