AI Compute Landscape: GPU Architectures, Tensor Cores, NVLink, and Scaling Challenges

AI algorithms can run on many types of chips—CPU, GPU, FPGA, NPU, ASIC—but their execution efficiency varies dramatically, and CPUs quickly lose performance when handling multiple tasks, making them unsuitable for AI computation.

Heterogeneous CPU + xPU solutions have become the standard for high‑performance AI workloads, with GPUs being the most widely used AI chip; in 2021, GPUs held an 89% market share in China’s AI chip market.

NVIDIA’s GPU evolution began with the 2006 release of CUDA, enabling general‑purpose GPU computing. The 2010 Fermi architecture introduced the first complete GPU compute architecture, followed by Kepler (adding FP64 and GPUDirect), Pascal (introducing NVLink), Volta (adding Tensor Cores), and later Turing, Ampere, and H100, each expanding support for lower‑precision data types such as INT8, FP16, TF32, and FP8.

Tensor Cores dramatically boost AI performance: Volta’s design delivers 64 FMA operations per clock (12× faster than Pascal), while A100’s Tensor Cores provide a 3× speedup over previous generations, and H100’s FP8 Tensor Cores achieve a 6× increase over A100’s FP16 performance.

Memory bandwidth is a critical bottleneck; GPUs adopt high‑bandwidth memory (HBM) stacked via TSV to overcome the “memory wall,” whereas consumer GPUs still rely on GDDR, which offers lower bandwidth and higher latency.

Inter‑GPU communication also limits scaling: PCIe bandwidth is constrained, prompting NVIDIA to develop NVLink and NVSwitch. NVLink’s fourth generation supplies 900 GB/s bidirectional bandwidth per GPU, and NVSwitch provides a fully connected mesh for up to 16 GPUs, reducing latency and simplifying cluster management.

Scaling AI models to billions of parameters (e.g., GPT‑3 with 175 B parameters) exposes the limits of single‑GPU training; estimates show that training on eight V100 GPUs would take 36 years, while a 512‑GPU cluster still requires about seven months, underscoring the necessity of large GPU clusters for modern AI research.