Unlocking GPU Server Architecture: PCIe, NVLink, NVSwitch & HBM Explained

GPU Server Fundamentals

Large‑scale model training uses servers that host eight high‑performance GPUs such as NVIDIA A100, A800, H100, H800 or future L40S models. The internal topology of an 8‑GPU A100 server is shown below.

PCIe Switch Chip

CPUs, memory modules, NVMe storage, GPUs and network adapters are interconnected via the PCIe (Peripheral Component Interconnect Express) bus or dedicated PCIe switch chips. PCIe Gen5 delivers up to 128 GB/s per direction, making it the backbone of modern high‑performance computing clusters.

NVLink Overview

Definition

NVLink is NVIDIA’s proprietary high‑speed point‑to‑point interconnect and communication protocol. It creates a mesh network that can link CPUs to GPUs or GPUs to GPUs, allowing multiple links per device. First released in March 2014, NVLink has evolved through four generations.

Evolution

Each generation increases the lane count and the total bidirectional bandwidth:

NVLink 1.0 : 4‑lane link, up to 160 GB/s total bidirectional bandwidth, primarily used to accelerate GPU‑GPU data transfer.

NVLink 2.0 : 6‑lane link, up to 300 GB/s total bidirectional bandwidth, providing higher transfer rates and improved communication efficiency.

NVLink 3.0 : 12‑lane link, up to 600 GB/s total bidirectional bandwidth, adds new protocols that further boost bandwidth and efficiency.

NVLink 4.0 : 18‑lane link, up to 900 GB/s total bidirectional bandwidth, meeting the bandwidth demands of AI and HPC workloads.

NVSwitch

NVSwitch is an NVIDIA‑designed switch chip that enables high‑speed, low‑latency communication among multiple GPUs within a single host. In a typical 8‑GPU A100 system (e.g., the Inspur NF5488A5), the NVSwitch chip sits beneath the six large heat sinks and directly connects all eight GPUs, ensuring efficient data exchange.

NVLink Switch

The NVLink Switch, introduced in 2022, is a standalone device that provides high‑performance GPU‑to‑GPU communication across multiple hosts. It differs from the integrated NVSwitch, which resides inside a single server.

HBM (High‑Bandwidth Memory)

Traditional GPU memory uses DDR modules accessed over PCIe, which can become a bandwidth bottleneck (PCIe Gen4 ≈ 64 GB/s, Gen5 ≈ 128 GB/s). High‑Bandwidth Memory (HBM) stacks multiple DDR dies vertically and places the memory directly on the GPU die, bypassing PCIe and delivering orders‑of‑magnitude higher throughput. GPUs such as the NVIDIA H100 integrate HBM.

HBM Generations

The evolution from HBM1 to HBM3e is illustrated below.

Bandwidth Unit Interpretation

Large‑scale GPU training performance depends on several bandwidth channels: PCIe, system memory, NVLink, HBM and network. Network speeds are expressed in bits per second (bit/s) and are usually reported per direction (TX/RX). PCIe, memory, NVLink and HBM bandwidths are reported in bytes per second (Byte/s) or transactions per second (T/s) and typically represent the total bidirectional capacity. Accurate conversion and comparison of these units is essential for evaluating data‑transfer capabilities that affect training efficiency.