Rising Compute Demand of Generative AI Models and GPU Accelerator Trends in 2024

Based on the reference "Model Transformation: Cloud Integration as a Trend (2024)", the intelligence level of GPT models has continuously improved from GPT‑1 to GPT‑5, with ChatGPT pre‑trained on 300 billion words and 1.75 trillion parameters (GPT‑2 had only 1.5 billion).

AI’s demand for cloud vendors’ capital expenditure began in Q4 2023; forecasts suggest North American cloud capex will return to a high‑growth trajectory in 2024.

Transformer compute demand has grown 750× in two years, roughly a ten‑fold increase per year. NVIDIA releases a new accelerator generation roughly every two years, each improving compute performance by about three times while price growth lags behind.

In generative AI scenarios, the compute required for training and inference is directly proportional to model parameters; larger models thus continuously increase capital spending.

Training: Compute scales with model parameters and dataset size (tokens). Inference: Compute scales with model parameters, response size, and traffic volume.

Future GPT‑5 is expected to reach 10 trillion parameters, further accelerating compute demand.

AI model compute needs have surged, with Transformer demand increasing nearly ten‑fold annually. GPT‑1 (2018) had 100 million parameters; GPT‑5 is projected to have 10 trillion.

Accelerator compute is the core performance metric. Standard training precision is FP32, though FP16 is sometimes used to save resources; inference often uses INT8 for higher efficiency.

NVIDIA remains the industry leader; its latest Blackwell GPUs surpass competitors in 8‑32‑bit inference compute and introduce FP4 for low‑precision scenarios.

AMD’s MI300X offers 1.3× the INT8/FP16/FP32 performance of NVIDIA H100 and comparable interconnect bandwidth (≈896 GB/s), making it attractive for cloud providers.

NVIDIA’s NVLink and NVSwitch upgrade roughly every two years, now delivering 1.8 TB/s bidirectional bandwidth, outpacing rivals. Competitors such as Google’s TPU v5p and Meta’s MTIA v2 also push interconnect and compute capabilities.

U.S. export controls on high‑end chips (A100, H100, etc.) have spurred domestic AI‑chip development (e.g., Ascend, Cambricon), accelerating AI‑chip localization.

NVIDIA’s product cadence (new generation every ~2 years) improves compute, memory, and interconnect roughly two‑fold each cycle. New training cards (H200, B200) and inference cards (L40, L40S, L20, L2, L4) offer higher HBM capacity and better price‑performance.

NVL 72’s copper interconnect reduces cost, power consumption, and failure rates compared to optical modules, supporting up to 576 GPUs across 8 racks with 1.8 TB/s bandwidth.

Overall, the rapid scaling of AI model parameters and the competitive acceleration hardware market are driving unprecedented growth in compute demand, influencing cloud capex, interconnect technology choices, and the push for domestic AI‑chip solutions.