Inside NVIDIA Hopper H100: Architecture, Performance, and AI Breakthroughs

At the 2022 NVIDIA GTC conference, the company unveiled the Hopper‑based H100 GPU, the most powerful accelerator for AI, high‑performance computing (HPC) and data analytics to date, named after computing pioneer Grace Hopper.

The H100 is fabricated on TSMC's 4 nm process (an optimized version of N5), with a die area of 814 mm²—14 mm² smaller than the A100—but a transistor count that jumps from 542 billion to 800 billion thanks to the denser process.

Architecturally, the Hopper GPU consists of eight Graphics Processing Clusters (GPCs). Every four GPCs share a 25 MB L2 cache, and each GPC contains nine Texture Processor Clusters (TPCs). Each TPC houses two Streaming Multiprocessors (SMs), and the entire chip connects to HBM3 memory with a 5120‑bit interface and up to 80 GB capacity.

Compute tasks arriving via PCIe 5.0 or NVLink are distributed by a Multi‑Instance GPU (MIG)‑controlled GigaThread engine to the GPCs. Within each GPC, the TPCs execute workloads using both CUDA cores and fourth‑generation Tensor Cores.

The Hopper design introduces a new thread‑block cluster mechanism that enables cross‑unit collaborative computation, improving scalability for large models. Each SM includes 128 FP32 CUDA cores and four Tensor Cores, supported by L1 and L0 instruction caches, a Wrap Scheduler, and a Dispatch Unit.

Crucially, Hopper adds FP8 Tensor Cores that support FP32/FP16 accumulators and two FP8 formats (E4M3 and E5M2). FP8 halves storage requirements and doubles throughput, especially when combined with the new Transformer engine, which dynamically switches precision per layer to maximize performance.

The Transformer engine, together with FP8 Tensor Cores, delivers up to 9× faster AI training and 30× faster inference on large NLP models compared to the previous generation.

Hopper also introduces the Tensor Memory Accelerator (TMA), an asynchronous data‑movement engine that offloads address‑generation and copy tasks from the compute threads, similar to a DMA controller, allowing threads to focus on computation.

Overall, the H100 achieves roughly a six‑fold increase in raw compute performance over the Ampere‑based A100, driven by the FP8 Tensor Cores, the Transformer engine, and the efficiency gains from TMA.