Inside Baidu’s High‑Performance GPU Cluster: Powering the Next‑Gen AI Models

1. Birth of Wenxin Yiyan

Wenxin Yiyan was trained on China’s largest AI‑focused high‑performance GPU cluster. Baidu Intelligent Cloud began planning this cluster in June 2021, collaborating with NVIDIA to design an InfiniBand (IB) network capable of housing more than ten thousand GPUs. The cluster was completed in April 2022, delivering EFLOPS‑level compute power.

2. High‑Performance Cluster Design

Beyond raw compute, the cluster required specialized design and optimization. Distributed training demands high‑throughput, low‑latency inter‑node communication via IB or RoCE, as well as carefully engineered intra‑node networking and topology to meet large‑model communication needs.

Different parallelism strategies (data, model, pipeline, expert, 4D hybrid) generate distinct communication patterns such as Allreduce, All2All, etc. Baidu therefore optimized both single‑node servers and the cluster network.

On the server side, Baidu’s X‑MAN 4.0 AI supercomputer provides 134 GB/s intra‑node Allreduce bandwidth and ranks top‑2 on the MLCommons 1.1 benchmark for comparable hardware.

The cluster network uses a three‑level Clos architecture optimized for large‑model training, reducing hop counts for same‑rank GPU communication and delivering high throughput with low latency. It currently supports up to 16 000 GPUs, achieving 98 % stable network performance and delivering a 3.87× training efficiency gain over the previous generation.

3. Challenges of Large‑Model Training

Model parameter counts have been growing tenfold annually, moving from billions to hundreds of billions of parameters. Training a 175 billion‑parameter GPT‑3‑scale model would take 32 years on a single A100 GPU, 34 days on 1 024 A100s at 45 % utilization, and exceeds single‑GPU memory capacity (700 GB required vs. 80 GB available).

These challenges manifest as three “walls”: the compute wall (gap between single‑GPU TFLOPS and total model FLOPs), the memory wall (insufficient GPU memory), and the communication wall (frequent parameter synchronization across GPUs). Overcoming each wall requires distributed training, but introduces new communication bottlenecks that can degrade scaling if not properly addressed.

4. End‑to‑End Training Process

The training workflow can be divided into two stages:

Parallel strategy and training optimization : After a model is submitted, the AI framework analyses the model structure and cluster capabilities to generate a parallel strategy and place AI tasks onto specific GPUs/XPUs. Optimizations include replacing standard operators with high‑performance equivalents and selecting communication strategies tailored to the strategy and network.

Resource management and task scheduling : The cluster provides the necessary compute, network, and storage resources, handling environment setup, data I/O, and inter‑GPU communication. Elastic fault‑tolerance and dynamic scheduling ensure long‑running jobs remain stable despite hardware failures or scaling changes.

Both stages rely on tight integration between the AI framework and the training cluster to break the three walls and ensure efficient, stable training.

5. Full‑Stack “AI Base” Acceleration

Ba​idu’s “AI Base” unifies three layers—chip, framework, and model—through custom technologies: Kunlun chips, PaddlePaddle framework, and the Wenxin large model. Two AI engineering platforms, the AI Mid‑Platform and Baidu Bai‑Ge heterogeneous computing platform, further improve efficiency.

The AI Mid‑Platform uses the framework to generate parallel strategies and manage the full training lifecycle. Baidu Bai‑Ge provides chip enablement, resource management, and task scheduling, enabling topology‑aware placement and elastic fault‑tolerance.

Key capabilities include:

Model splitting (data, model, pipeline, expert, 4D hybrid) to overcome compute and memory walls.

Topology‑aware placement that maps tasks to GPUs based on intra‑node NVSwitch or inter‑node IB/RoCE links.

Automatic parallelism that searches optimal model partitioning and hardware mapping.

End‑to‑end adaptive training that re‑optimizes placement when the cluster changes, providing elastic scaling and fault‑tolerant recovery.

AI acceleration suite (AIAK) that optimizes data loading, operator execution, and distributed communication, achieving up to 90 % multi‑GPU acceleration on thousand‑GPU clusters.

Ba​idu Vice President Hou Zhenyu: “Large‑model training is a systems engineering effort; without full‑stack optimization, it is hard to ensure smooth training. Our complete software stack accelerates large‑model training by up to 2.1×.”

6. Democratizing AI in the Large‑Model Era

To make large‑model capabilities widely accessible, Baidu launched the Yangquan AI Computing Center in late 2022, offering 4 EFLOPS of heterogeneous compute—the largest single data center in Asia. The AI Base is now open to the public through various delivery models (regional cloud, edge cloud, local clusters, private cloud), enabling enterprises and developers to obtain AI services easily.

Overall, Baidu’s integrated hardware, framework, and resource‑management stack demonstrates how a full‑stack, system‑level approach can break the compute, memory, and communication walls that traditionally limit large‑model training.