How to Decode AI Chip Compute Units and Avoid Common Pitfalls

Why a single GPU reports multiple compute numbers

Different numbers correspond to different operating modes, precisions, or sparsity settings.

Two basic units

FLOPS measures floating‑point operations per second and includes FP64, FP32, TF32, FP16/BF16, FP8, and FP4. 1 PFLOPS = 1000 TFLOPS.

TOPS measures integer operations per second (e.g., INT8, INT4) and is expressed as tera‑operations per second.

Example: RTX 5090 theoretical peaks

FP32 : 104 TFLOPS

FP16/BF16 (Tensor Core) : 1614 TFLOPS ≈ 1.61 PFLOPS

FP8 (Tensor Core) : ≈ 3.3 PFLOPS

FP4 (Tensor Core) : ≈ 6.6 PFLOPS

INT8 : ≈ 3300 TOPS

INT4 : ≈ 6600 TOPS

All values are derived from the same Tensor‑Core hardware; they are not additive.

Precision vs. reported throughput

Halving the data‑bit width halves the amount of data processed per operation, effectively doubling the number of operations that can be issued each clock cycle. Consequently, manufacturers can present larger PFLOPS numbers for lower‑precision modes (FP8, FP4). Training typically uses higher precision (BF16/FP8) for numerical stability, while inference often selects the lowest precision that meets accuracy requirements.

Structured sparsity as a third variable

Enabling 2:4 structured sparsity can double the theoretical peak. For NVIDIA H100 SXM5 the dense FP16 Tensor‑Core peak is 989.5 TFLOPS; with sparsity enabled the peak becomes 1979 TFLOPS. The boost assumes the model weights are sparsified accordingly, which is uncommon in production.

Roofline model: compute‑bound vs. bandwidth‑bound

The Roofline model relates arithmetic intensity (FLOPs ÷ bytes moved) to achievable performance. Large matrix multiplications have high intensity and are compute‑bound, allowing GPUs to approach their peak FLOPS. Token‑wise LLM inference has low intensity and is often bandwidth‑bound, limiting actual throughput despite high PFLOPS.

Theoretical peak vs. real‑world utilization (MFU)

Manufacturers quote theoretical peaks; actual workloads achieve only a fraction. Model FLOPs Utilization (MFU) = effective FLOPs ÷ theoretical peak. Reported MFU values range from 40‑55 % for leading labs training massive models to 20‑30 % for many teams, limited by memory bandwidth, multi‑GPU communication, kernel efficiency, hyper‑parameter choices, or data‑loading bottlenecks.

Practical monitoring

Use nvidia-smi to view GPU utilization, but GPU‑Util only indicates that at least one core was active. For finer diagnostics run: nvidia-smi dmon -s pucvmet -d 1 Check the sm (SM utilization) and mem (memory controller utilization) columns. High sm with low mem suggests a compute‑bound workload; high mem indicates a bandwidth bottleneck. Verify that clocks.current.sm equals clocks.max.sm to confirm the GPU is not throttled.

Checklist when interpreting a compute figure

Is the metric FLOPS (floating‑point) or TOPS (integer)?

What precision does the number represent? Different precisions differ by orders of magnitude.

Is the figure for a dense or sparsified configuration? Sparse peaks are typically twice the dense value.

Is the number a theoretical peak or an observed throughput? Observed throughput is reduced by MFU.

Is the workload compute‑bound or bandwidth‑bound? Large‑model inference is often limited by memory bandwidth rather than raw PFLOPS.

Key takeaways

FLOPS/PFLOPS describe floating‑point capability; TOPS describes integer capability. A single GPU can report multiple peaks because the same hardware operates under different precisions, modes, or sparsity settings. These peaks cannot be summed or directly converted. Distinguish dense versus sparse peaks and recognize the gap between theoretical peaks and real‑world MFU. Effective AI training and inference depend on memory capacity, bandwidth, and interconnect performance in addition to raw compute.