How AI Servers Connect: Inside PCIe, NVLink, and Memory Interfaces

According to TrendForce, AI server shipments reached about 130,000 units in 2022, roughly 1% of global server shipments, and are expected to grow at 15.4% YoY in 2023, driven by products from Microsoft, Meta, Baidu and ByteDance, with a projected CAGR of 12.2% through 2027.

The latest NVIDIA DGX H100 system, released in 2022, integrates eight H100 GPUs (640 billion transistors) and delivers six‑times higher AI performance at FP8 precision, offering 900 GB/s bandwidth via PCIe 5.0 and NVLink interconnects.

PCIe Switch and Retimer Technology

PCIe switches act as PCIe‑compatible bridges that aggregate multiple device links, expanding the number of PCIe lanes available to CPUs, GPUs and other accelerators. The market for PCIe chips was $790 million in 2021 and is forecast to reach $1.8 billion by 2028 (CAGR 11.9%). China is the largest market, driven by high‑performance computing, cloud and AI workloads.

Retimer chips ensure signal integrity for CPU‑GPU links; typical AI servers use several retimers, e.g., Astera Labs provides four per system. Leading retimer vendors include Spectrum‑KY, Astera Labs and Lattice (formerly Unisoc), with additional competition from Renesas, Texas Instruments and Microchip.

GPU‑to‑GPU Interconnects

NVIDIA’s NVLink technology has evolved from the first generation (40 GB/s per link, Pascal GP100) to the fourth generation (50 GB/s per link, Hopper architecture) and now supports up to 18 links per GPU, delivering up to 900 GB/s total bandwidth. NVSwitch, introduced in 2018 and now in its third generation, provides 64 NVLink 4.0 ports per switch and enables up to 900 GB/s GPU‑to‑GPU communication.

Competing high‑speed interconnects include AMD’s Infinity Fabric and Intel’s CXL, which also target CPU‑GPU and GPU‑GPU communication.

Memory Interface Chips

Server memory modules (RDIMM, LRDIMM) rely on memory interface chips—register‑clock‑driver (RCD) and data‑buffer (DB) devices—to buffer address/command signals and data streams, respectively. DDR4 memory interface chips have progressed to 3200 MT/s (Gen 2plus), while DDR5 chips are defined for 4800, 5600 and 6400 MT/s, with higher‑speed sub‑generations expected.

The DDR5 memory‑interface market was $280 million in 2016 and grew to $570 million by 2018 (CAGR 40%). Only three vendors—Lattice, Renesas (IDT) and Rambus—can currently mass‑produce first‑generation DDR5 interface chips, creating a high barrier to entry.

For further technical details on InfiniBand, RoCE, and other high‑performance networking technologies, see the referenced articles.