Choosing the Right AI Data Center Network: InfiniBand vs RoCE

Background

Previous articles introduced AI Data Center (AIDC) concepts and compared them with traditional IDC. As model parameters grow, training demands increase compute and memory, e.g., GPT‑3 with billions of parameters requires 2 TB of GPU memory, which exceeds single‑card capacity.

Need for Distributed Training

To reduce training time from years to days, multi‑node, multi‑GPU parallelism using distributed training is required.

Challenges: Compute and Storage Walls

Large‑scale AI training faces compute‑wall and storage‑wall challenges, requiring clusters with massive compute and memory capabilities.

Network's Role

The high‑performance network connecting the cluster determines inter‑node communication efficiency, affecting overall throughput. Desired network characteristics include low latency, high bandwidth, long‑term stability, massive scalability, and operability.

Network Options for AIDC

Two main large‑scale network architectures are used: InfiniBand and RoCE.

InfiniBand

InfiniBand is designed for high‑performance computing, with mainstream 400 Gbps NDR technology. It offers native lossless networking via credit‑based flow control, preventing buffer overflow and packet loss. Adaptive Routing provides per‑packet dynamic routing, supporting massive GPU clusters.

RoCE (RDMA over Converged Ethernet)

RoCE brings RDMA to Ethernet, leveraging a mature Ethernet ecosystem. Advantages include an open ecosystem, higher raw rates (1 Gbps–800 Gbps, future 1.6 Tbps), lower cost due to commodity Ethernet switches, and simpler deployment and maintenance.

Business‑Level Comparison

Performance: InfiniBand’s lower end‑to‑end latency gives better application performance, though RoCEv2 meets most AI workloads.

Scale: InfiniBand supports up to ten‑thousand‑GPU clusters without performance loss; RoCEv2 typically supports up to a thousand GPUs.

Operations: InfiniBand offers more mature multi‑tenant isolation and diagnostics.

Cost: RoCEv2 is generally cheaper due to lower Ethernet switch prices.

Vendors: InfiniBand is dominated by NVIDIA, while RoCEv2 has a broader vendor base.

Network Design for AIDC

To meet AI training bandwidth demands, clusters often use 8‑GPU nodes with multiple 100 Gbps NICs, sometimes employing NVLink+NVSwitch. High‑bandwidth design uses Fat‑Tree topology with non‑blocking 1:1 uplink/downlink ports.

Low‑latency AI‑Pool design groups 8 nodes, allowing same‑index GPUs to communicate via a single hop using NVSwitch and NCCL RailLocal.

Fat‑Tree architectures can be two‑layer or three‑layer. A two‑layer Fat‑Tree with 40‑port switches can support up to 800 GPUs; a three‑layer Fat‑Tree can scale to 16 000 GPUs.