How Multi‑Chip Heterogeneous Clusters Power Next‑Gen Large Model Training

Introduction

The article opens with a story from the novel "The Legend of the Condor Heroes" to illustrate how a group of disciples (GPUs) must form a coordinated formation to defeat a powerful boss, setting the stage for discussing large‑model training on massive GPU or mixed‑chip clusters.

Why Massive Clusters Are Needed

Training state‑of‑the‑art models such as GPT‑4 or the upcoming GPT‑5 demands tens of thousands of GPU cards; a single GPU or a small cluster cannot handle the workload. Supply‑chain constraints and high costs make it difficult to assemble a full set of GPUs, prompting the exploration of heterogeneous clusters that combine domestic chips with GPUs for better cost‑performance.

Three Key Components of a High‑Performance Cluster

Inter‑connect and communication : Physical connections (NVLink within a server, InfiniBand or RoCE between servers) and communication libraries (NCCL for NVIDIA GPUs, XCCL/HCCL for Kunlun and Ascend chips) enable data exchange across thousands of cards.

Distributed parallel strategy : The workload must be split among all devices using data parallelism, pipeline parallelism, tensor parallelism, or combinations thereof, ensuring each GPU has useful work.

Accelerator (software) stack : Optimizations such as data prefetching, custom operators, and enhanced collective communication improve per‑card performance and overall efficiency.

Baidu Baige Solution for Heterogeneous Training

To achieve cross‑chip inter‑connect, Baidu Baige uses CPU‑forwarding to bridge Ascend sub‑clusters with GPU sub‑clusters and introduces a custom collective library called BCCL, an enhanced version of NCCL that works across GPUs, Kunlun, and Ascend devices.

Baige also implements an adaptive parallel‑strategy search powered by an AI‑efficiency matrix that records compute, memory, and I/O characteristics of each chip, automatically selecting the optimal mix of data, pipeline, and tensor parallelism for a given heterogeneous topology.

Finally, Baige provides an "Accelerator Abstraction Layer" that decouples chip‑specific operators from higher‑level training strategies, allowing chip vendors to focus on kernel optimization while Baige reuses proven strategies across all hardware.

Performance Results

Using Baige’s stack, Baidu reports mixed‑chip clusters achieving 97% efficiency for a hundred‑card setup and 95% for a thousand‑card setup, effectively delivering near‑full performance despite heterogeneous hardware.

The approach can be delivered via Baidu Intelligent Cloud public services or on‑premises private clouds, offering a cost‑effective solution to the chronic GPU shortage while enabling large‑model training without sacrificing performance.