Why NVLink Beats PCIe for AI Training: A Deep Dive into GPU Interconnects

GPU Network Interconnect Technologies

In AI compute centers, both Scale‑Out (inter‑node) and Scale‑Up (intra‑node) interconnects are essential. Scale‑Out handles communication between servers, while Scale‑Up connects accelerators within a single server.

Scale‑Out Technologies

Ethernet (UEC/Ultra‑Ethernet) : Low‑cost, widely deployed, supports RDMA and high‑bandwidth modes for AI clusters.

InfiniBand : Provides ultra‑low latency (microseconds) and high bandwidth for multi‑node HPC and AI workloads.

RoCE (RDMA over Converged Ethernet) : Implements RDMA over standard Ethernet, reducing CPU overhead and latency while keeping hardware costs lower than pure InfiniBand.

Scale‑Up Technologies

PCIe : Traditional high‑speed serial bus used for GPU‑CPU connections; bandwidth limited by lane count (e.g., PCIe 5.0 x16 ≈ 128 GB/s) and suffers from latency and contention at large GPU counts.

CXL : Cache‑coherent extension of PCIe, enabling shared memory between CPU and accelerators.

NVLink : NVIDIA’s proprietary high‑bandwidth, low‑latency link (up to 900 GB/s in NVLink 4.0, 1.8 TB/s in NVLink 5.0) with full cache‑coherency, ideal for multi‑GPU training.

NVSwitch : Switches multiple GPUs into an all‑to‑all topology, delivering up to 1.8 TB/s bi‑directional bandwidth and 130 TB/s aggregate bandwidth.

UALink (Ultra Accelerator Link) : Open, Ethernet‑based accelerator interconnect (200 GT/s per lane, up to 800 GT/s aggregate) supporting up to 1024 accelerators, with features like hardware encryption, multi‑tenant isolation, and deterministic latency.

UB‑Mesh (Huawei) : Unified mesh network that aggregates up to 1024 NPUs in a full‑mesh topology, scalable to 8000 nodes, offering 10 Tbps per link and sub‑microsecond latency.

Proprietary Vendor Links : AWS NeuronLink (PCIe‑based), Google ICI (custom programmable interconnect for TPUs), and Broadcom SUE (Scale‑Up Ethernet) provide vendor‑specific high‑performance paths.

Comparison and Trade‑offs

NVLink offers the highest bandwidth and lowest latency for GPU‑GPU communication but is expensive and proprietary. PCIe is universal and cost‑effective but limited in bandwidth for large clusters. InfiniBand excels in multi‑node scaling with low latency, while RoCE offers a cost‑effective RDMA alternative over Ethernet. UALink and UB‑Mesh aim to provide open, high‑bandwidth, low‑latency fabrics that can scale to thousands of accelerators without the licensing constraints of NVLink.

Future Directions

PCIe 6.0/7.0/8.0 promise higher raw speeds (up to 256 GB/s per x16 lane), but architectural bottlenecks remain. Emerging standards like UALink 1.0, UB‑Mesh, and Broadcom’s SUE target unified, open interconnects that combine Ethernet compatibility with accelerator‑grade performance, potentially reshaping AI data‑center networking in the next decade.

Overall, selecting the right interconnect depends on workload characteristics, budget, and scalability requirements. For single‑node, GPU‑intensive training, NVLink/NVSwitch remains dominant, while large‑scale multi‑node clusters often combine InfiniBand or RoCE for Scale‑Out and PCIe or emerging open fabrics for Scale‑Up.