How Baidu’s BCCL Boosts Large‑Model Training with Real‑Time Observability and Fault Diagnosis

1. Collective Communication Is Essential for Distributed Training

In distributed training each GPU processes only a portion of the model or data. GPUs synchronize gradients and update parameters through collective communication, allowing the whole cluster to work as a single accelerator.

If a single GPU stalls during collective communication, all other GPUs must wait, slowing the entire job.

Therefore, collective‑communication performance directly determines the speed of distributed tasks and the ability of the cluster to accelerate model training.

2. Large Models Impose New Operational and Stability Requirements

Training large models can last weeks, with clusters scaling to thousands or tens of thousands of GPUs. Over such long periods many failures occur, reducing resource utilization or causing job interruptions.

Insufficient operational reliability shortens the effective training time and inflates costs—for example, a 30‑day job may lose 10 days to fault handling, which is unacceptable.

The collective‑communication library, as a core system component, must therefore be optimized for operability and stability in large‑model scenarios.

3. Overview of Baidu Collective Communication Library (BCCL)

BCCL is Baidu Intelligent Cloud’s communication library tailored for large‑model training and a key component of the Baidu Baige 3.0 platform. Built on the open‑source NCCL, BCCL adds functionality and enhances capabilities in observability, fault diagnosis, and stability, while also optimizing performance for Baidu’s custom GPU chips.

Observability: real‑time bandwidth statistics.

Fault diagnosis: detection of hangs in collective communication.

Stability: improved network resilience and fault‑tolerance.

Performance optimization: higher bandwidth utilization on mainstream GPU chips.

The following sections detail these four capabilities.

4. Observability – Real‑Time Bandwidth Statistics

4.1 Background

During training, overall cluster performance may degrade without obvious causes, requiring operators to inspect all components.

4.2 Problem

Existing monitoring platforms cover storage, RDMA, and GPUs but lack direct, real‑time metrics for collective‑communication performance. Currently, engineers resort to indirect RDMA traffic monitoring or stop training to run nccl‑test, both of which are disruptive.

4.3 Features and Effects

BCCL provides real‑time bandwidth statistics for collective communication, accurately showing performance at each training stage. This data supports fault isolation, performance tuning, and verification of whether bandwidth reaches hardware limits.

5. Fault Diagnosis – Identifying Communication Failures

5.1 Background

GPU failures can cause training jobs to hang without generating explicit error logs, leading to “silent faults” that appear only after hours or days of execution.

5.2 Problem

Because collective communication is synchronous, a failing GPU leaves other GPUs waiting, and no log is emitted. Traditional tools like nccl‑test cannot reliably reproduce hangs.

5.3 Features and Effects

BCCL continuously records internal communication states. When a hang occurs, it outputs per‑rank status, enabling engineers to narrow down the faulty GPU with minimal disruption, dramatically reducing diagnosis time from days to minutes.

6. Stability – Network Resilience and Fault Tolerance

6.1 Background

Transient network port up‑downs can abort training tasks, but such physical glitches are inevitable.

6.2 Features and Effects

Control‑plane fault tolerance: retry mechanisms during task startup to survive occasional network failures.

Data‑plane fault tolerance: improved RDMA retransmission limits to prevent crashes during normal operation.

7. Performance Optimization – Enhancing Collective‑Communication Throughput

Even on mainstream GPU chips, there is room to increase collective‑communication bandwidth for large‑model workloads.

In a dual‑node H800 test environment, BCCL achieved a 10 % higher bandwidth utilization compared with NCCL.

8. Summary

On December 20, 2023, Baidu released the Baige AI Heterogeneous Computing Platform 3.0, a smart infrastructure optimized for large models. With BCCL’s operational and stability enhancements, the platform achieves 98 % effective training time and 95 % bandwidth utilization.