How AI Model Scaling is Driving a GPU and Cloud Compute Arms Race in 2024

Evolution of Large‑Language Models

The parameter count of GPT‑style models has grown from 100 M (GPT‑1) to 175 B (GPT‑3) and is projected to reach 10 trillion for GPT‑5. Correspondingly, training data volume increased from a few gigabytes to tens of terabytes.

AI‑Driven Cloud Capital Expenditure

Since Q4 2023 AI workloads have become a primary driver of cloud providers’ cap‑ex. Forecasts indicate a resurgence of high‑growth spending in North America in 2024 as vendors provision more GPU‑rich infrastructure for training and inference.

Compute Demand Growth

Transformer compute requirements have risen ~750× over the past two years, roughly a ten‑fold increase per year. Each new accelerator generation (e.g., NVIDIA A‑series → B‑series) typically delivers a ~3× performance boost while price growth lags.

Training vs. Inference Compute Scaling

Training: Compute scales linearly with model parameters, token count, and dataset size.

Inference: Compute scales with model parameters, output length, and request volume.

Accelerator Landscape

NVIDIA: Blackwell platform leads with superior 8‑32‑bit inference performance and introduces FP4 for ultra‑low‑precision inference. NVLink bandwidth reaches 1.8 TB/s bidirectional; the GB200 NVL72 rack can interconnect up to 576 GPUs.

AMD: MI300X surpasses NVIDIA in FP64 performance and offers 1.3× INT8/FP16/FP32 throughput of the H100, with up to 896 GB/s inter‑GPU bandwidth.

Google: TPU v5p doubles floating‑point throughput and triples memory bandwidth versus TPU v4, with inter‑chip bandwidth up to 600 GB/s.

Other custom accelerators include Meta MTIA v2, Microsoft Maia 100 (Azure), and Amazon Trainium 2 / Graviton 4.

Market Share and Competition

TechInsights reports NVIDIA held 98 % of global data‑center GPU shipments in 2023, but AMD, Google, Tesla and others are gaining traction.

Domestic (China) Market Dynamics

U.S. export controls (Oct 2023) placed chips such as A100, H100, L40 on a restricted list. Chinese vendors (Ascend, Cambricon) are accelerating AI‑chip development, driving a trend toward domestic chip localization.

NVIDIA Product Cadence and Pricing

Since 2020 NVIDIA releases a new generation roughly every two years, each improving compute and interconnect bandwidth ~2× with modest price increases. The H200 accelerator expands HBM capacity from 80 GB to 141 GB while maintaining a strong price‑performance ratio.

Training and Inference Cards in 2024

Training cards: H100, B100, and the newer H200/B200 series provide higher performance‑per‑dollar, especially in memory bandwidth.

Inference cards: L40/L40S dominate 2024 shipments; newer L20, L2, and L4 models address varied workloads.

Interconnect Technologies

NVIDIA’s NVLink and NVSwitch follow a two‑year upgrade cycle; NVLink now delivers 1.8 TB/s bidirectional bandwidth. Competing solutions include AMD’s 896 GB/s and Google’s 600 GB/s interconnects. Most other cloud providers still rely on PCIe, which offers lower bandwidth.

Copper‑Based Interconnect Advantages

The GB200 NVL72 rack uses high‑speed copper cables, which lower cost, reduce power consumption, and improve reliability compared with optical modules. Copper interconnects also lower system‑level costs and enhance compute‑price efficiency.

Scalable Rack Architecture

Copper cabling enables dense GPU clusters: up to 576 GPUs can be interconnected across eight racks, with compute‑tray and rack‑to‑rack copper links providing high bandwidth and low fault rates.