Harvard GPU Lecture Notes: Roofline Analysis of LLM Inference on Hopper & Blackwell

GPU Fundamentals: Throughput-Oriented Hardware

GPUs sacrifice single-thread performance for massive parallelism. The NVIDIA H200 SXM exemplifies this with 132 SMs, 16,896 FP32 cores, and 270,000+ concurrent threads. Warp-level scheduling (32 threads per warp) hides memory latency by switching to ready warps when a warp stalls on cache misses. SIMT (Single Instruction Multiple Threads) executes one instruction across 32 threads; branch divergence within a warp serializes execution paths, reducing throughput.

Three Hard Limits Governing LLM Performance

Peak Compute : Theoretical BF16/FP8 throughput caps matrix-multiplication speed.

HBM Memory Bandwidth : Limits weight and KV-cache read rates.

VRAM Capacity : Determines maximum model size and KV-cache length.

Roofline Model for LLM Kernels

The model plots attainable FLOP/s against arithmetic intensity (total FLOPs / bytes accessed). The ridge point separates two regimes:

Memory-bound (intensity < ridge) : Performance scales with HBM bandwidth.

Compute-bound (intensity > ridge) : Performance saturates at peak FLOP/s.

Prefill vs. Decode: Divergent Bottlenecks

Prefill processes the input prompt as large matrix multiplications (GEMM). High arithmetic intensity makes it compute-bound .

Decode generates tokens one by one via matrix-vector operations (GEMV). Weights are read once per token, yielding very low arithmetic intensity; it is memory-bound . On H200, single-batch decode achieves <1% of peak compute.

Batching raises decode arithmetic intensity by reusing weights across requests, yet KV-cache reads remain low-intensity. Longer contexts amplify KV-cache traffic. Industry trend: each GPU generation increases compute faster than bandwidth, pushing more kernels into the memory-bound region. Lower precision (FP8/FP4) shifts the ridge point toward higher arithmetic intensity.

H200 Architecture Deep Dive

132 SMs share 50 MB L2 cache and 141 GB HBM3e. Each SM contains four partitions, each with warp schedulers, registers, FP32 units, and Tensor Cores . Tensor Cores execute matrix-multiply-accumulate (MMA) instructions; over 99% of LLM floating-point ops run on them, delivering massive BF16/TF32 throughput. Each SM hosts up to 64 resident warps (2,048 threads).

Memory Hierarchy

Registers → Shared Memory / L1 (256 KB per SM) → L2 (50 MB) → HBM → Host memory / other GPUs via NVLink or PCIe.

Multi-GPU Scaling

Scale-up (intra-node) : HGX H200 8-GPU servers use NVSwitch for full-mesh NVLink interconnect.

Scale-out (inter-node) : NVL72 rack-scale systems connect many GPUs through NVLink switches. Bandwidth hierarchy: HBM ≫ NVLink ≫ RDMA network ≫ PCIe.

NVIDIA Architecture Evolution

2006–2017 : Tesla → Fermi → Kepler → Maxwell → Pascal → Volta. Volta introduced Tensor Cores .

2018–2026 : Turing → Ampere → Hopper → Blackwell → Rubin. Focus on AI low-precision; compute growth far outpaces memory bandwidth, worsening the memory wall.

Competitor AI Accelerators

AMD MI355X : Chiplet design with Infinity Cache; 64-thread wavefront replaces 32-thread warp; ROCm/HIP software stack.

Google TPU Ironwood : Systolic-array cores with 3D torus interconnect; programmed via JAX / PyTorch/XLA.

AWS Trainium3 : Software-managed SRAM, no hardware cache; Neuron SDK and NKI kernel interface.

Specialized architectures : Cerebras (wafer-scale), Groq LPU, Microsoft Maia 200 — all leverage massive on-chip SRAM to reduce HBM dependence.

Key Takeaways

GPU strengths: massive thread parallelism, warp-switched latency hiding, Tensor Core matrix acceleration. Hard limits: peak compute, HBM bandwidth, VRAM capacity.

LLM inference phases differ fundamentally: prefill is compute-bound; decode is memory-bound with extremely low compute utilization. Batching, quantization, and KV-cache traffic reduction are primary decode optimizations. Newer chips widen the compute/bandwidth gap.

Hopper added TMA (Tensor Memory Accelerator) for asynchronous data movement; Blackwell adds TMEM (Thread Block Memory) and dual-SM cooperative MMA, further optimizing memory transport. Multi-GPU relies on NVLink for scale-up and RDMA for scale-out.

Beyond NVIDIA, AMD, Google, AWS, Cerebras, Groq, and Microsoft offer diverse architectures with different memory-organization and programming-model trade-offs; several use huge on-chip SRAM to alleviate external HBM pressure.

The Roofline model remains the essential analytical tool: arithmetic intensity instantly reveals whether a kernel is compute- or memory-bound.