How Baidu’s High‑Performance GPU Cluster Powers the Next Generation of Large‑Scale AI Models

1. Birth of Wenxin Yiyan

Wenxin Yiyan was trained on the nation’s largest AI‑focused high‑performance GPU cluster.

In June 2021, Baidu Intelligent Cloud began planning a new high‑performance GPU cluster to meet future large‑model training needs, collaborating with NVIDIA to design an IB network capable of housing over ten thousand GPUs. The cluster was completed in April 2022, delivering EFLOPS‑level compute.

In March 2023, Wenxin Yiyan was launched on this cluster and continues to scale.

NVIDIA China Solutions and Engineering GM Dr. Lai Junjie: "High‑speed IB‑connected GPU clusters are the key infrastructure for the large‑model era. The GPU/IB cluster built by NVIDIA and Baidu Intelligent Cloud is the largest high‑performance GPU/IB cluster in the domestic cloud market, accelerating Baidu’s breakthroughs in large models."

2. High‑Performance Cluster Design

High‑performance clusters require specialized design and optimization beyond simply aggregating compute power.

Distributed training demands high‑throughput, low‑latency inter‑node communication via IB, RoCE, and careful topology design to meet large‑model communication requirements.

Different parallel strategies (data, model, expert, 4D hybrid) generate varying communication patterns such as Allreduce, All2All, etc.

Baidu optimized both single‑node servers and cluster networking.

On the server side, Baidu’s X‑MAN 4.0 provides 134 GB/s intra‑node Allreduce bandwidth and ranks top‑2 on the MLCommons 1.1 leaderboard.

On the network side, a three‑tier Clos architecture minimizes hop counts for same‑GPU communication, supporting up to 16 000 GPUs with 98 % stable performance, achieving 3.87× training efficiency over the previous generation.

However, building a massive heterogeneous cluster is only the first step; systematic hardware‑software co‑optimization is also required.

3. Challenges of Large‑Model Training

Model parameter scales have been growing tenfold annually, reaching billions of parameters by 2020 and trillions by 2022.

Training such models now requires hundreds of servers and thousands of GPUs, extending training cycles from days to months.

For example, training a 175 billion‑parameter GPT‑3 would take 32 years on a single A100 at half‑precision; even 1 024 A100s would need 34 days, and a single GPU cannot hold the model in memory.

Three major “walls” impede training: the compute wall (single‑GPU compute vs total model compute), the memory wall (GPU memory insufficient for model parameters), and the communication wall (frequent parameter synchronization across GPUs).

These walls become more pronounced as model and cluster sizes increase, and hardware failures can further disrupt long‑running training jobs.

4. Process of Large‑Model Training

From an infrastructure perspective, training consists of two stages:

Stage 1: Parallel strategy and training optimization

The AI framework selects an appropriate parallel strategy based on model structure and cluster capabilities, placing AI tasks across GPUs/XPUs and optimizing data loading, operators, and communication.

Stage 2: Resource management and task scheduling

The training cluster provides high‑performance resources, manages storage, and schedules tasks, while also offering elasticity and fault tolerance to maintain training continuity.

5. Full‑Stack Fusion: The “AI Base” Accelerates Large‑Model Training

Leveraging years of AI and large‑model experience, Baidu introduced the self‑developed “AI Base” stack (chip‑framework‑model) comprising Kunlun chips, PaddlePaddle, and the Wenxin large model.

Two AI engineering platforms—AI Middle Platform and Baidu Baige AI Heterogeneous Computing Platform—enhance development and resource efficiency, breaking the three walls.

The AI Middle Platform uses the AI framework to devise parallel strategies and optimize the training environment throughout the model lifecycle.

Baige provides high‑performance chip enablement, resource management, and task scheduling.

The AI Base achieves end‑to‑end optimization and acceleration for large‑model training.

Baidu Vice President Hou Zhenyu: “Large‑model training is a systems engineering challenge. Without full‑stack optimization, it is difficult to ensure successful training. Our complete software stack accelerates large‑model training.”

5.1 Parallel Strategy and Training Optimization

PaddlePaddle supports diverse parallel strategies (data, model, pipeline, expert, 4D hybrid) to train models from billions to trillions of parameters, breaking compute and memory walls.

Baige’s topology‑aware capabilities sense intra‑node and inter‑node architectures, informing optimal GPU/XPU placement.

Automatic parallelism automatically determines optimal model partitioning and hardware mapping, placing tasks to maximize performance.

End‑to‑end adaptive training adjusts model partitioning and task placement in response to cluster changes, achieving up to 2.1× performance gains.

Training optimizations, including data loading, operator acceleration, and communication enhancements, yield up to 90 % multi‑GPU acceleration on thousand‑GPU clusters.

In MLPerf Training v2.1 (Nov 2022), Baidu’s PaddlePaddle + Baige solution ranked first worldwide for the same GPU configuration, surpassing NGC PyTorch in both training time and throughput.

5.2 Resource Management and Task Scheduling

Baige runs AI tasks on the CCE container engine, providing resource management, topology awareness, and elastic fault tolerance.

It offers compute, network, and storage resources such as Baidu Taihang elastic bare‑metal servers, IB/RoCE networks, parallel file systems, and object storage.

Elastic RDMA reduces communication latency by 2–3×, while high‑performance servers accelerate computation.

ECCL, Baidu’s heterogeneous collective communication library, detects slow or faulty nodes, enabling the cluster to re‑allocate tasks and maintain smooth training.

6. AI Democratization in the Large‑Model Era

Large models are a milestone toward artificial general intelligence; massive compute and full‑stack software optimization are essential to harness them.

In late 2022, Baidu launched the Yangquan Intelligent Computing Center, delivering 4 EFLOPS of heterogeneous compute—the largest single data center in Asia.

All AI Base capabilities are now open to the public via cloud, edge, local, and private deployments, enabling industries to access advanced AI services.