Why RTX 4090 Beats H100 for LLM Inference but Fails at Training

GPU Comparison for Large‑Model Training and Inference

The H100, A100 and RTX 4090 differ mainly in communication bandwidth and memory capacity, while raw FP16 compute is comparable (H100 1979 TFLOPS, A100 312 TFLOPS, 4090 330 TFLOPS). Memory sizes are 80 GB for H100/A100 and 24 GB for 4090; bandwidths are 3.35 TB/s, 2 TB/s and 1 TB/s respectively. Prices range from $30‑40 k for H100, $15 k for A100 to $1.6 k for 4090.

Why 4090 Cannot Train Large Models

Training large language models requires high‑speed inter‑GPU communication, large activation memory, and data‑center‑grade licensing. The 4090’s PCIe Gen4 bandwidth (≈64 GB/s) and 24 GB memory are far below the needs of a 70 B‑parameter model, whose activation, gradient and optimizer states exceed 1.8 TB. Consequently, even with tensor, pipeline or data parallelism, the communication overhead dominates and the GPU quickly runs out of memory.

Training Cost and Parallelism Analysis

Using the formula Flops = 6 × parameter‑count × token‑count, LLaMA‑2 70B needs about 1.7 M GPU‑hours on A100. To finish within a month, roughly 2 400 A100 GPUs are required. With 4090, the FP16 compute is similar to A100, but half the memory bandwidth and one‑third the memory capacity, so a single 4090 is slower and would need at least 2 048 GPUs, making communication the bottleneck. The three parallelism dimensions—tensor, pipeline and data parallelism—must be balanced; excessive pipeline stages explode activation memory, while fine‑grained tensor parallelism overwhelms the limited PCIe bandwidth.

Inference Advantages of 4090

Inference does not store gradients or optimizer states, only model parameters and KV‑Cache. KV‑Cache reduces repeated K/V computation, saving up to 16 K FLOPs per byte stored. For batch‑size 1 and short contexts, the workload becomes memory‑bound; the 4090’s 1 TB/s bandwidth yields a compute‑to‑bandwidth ratio of 330, meaning it is efficient up to ~330 tokens before bandwidth limits appear. With batch sizes around 330, compute and bandwidth are balanced, achieving ~330 tokens per second per GPU.

Cost‑Performance Comparison

Eight × 4090 cards cost ≈$12.8 k, plus $20 k for a server and networking, totalling about $32 k. Assuming a three‑year depreciation and $0.1/kWh electricity (≈5 kW total), the hourly operating cost is ~$2. This setup can generate roughly 44 M tokens per hour, i.e., $1 ≈ 22 M tokens. An eight‑card H100 system costs ≈$240 k, hourly cost ~$12, but can produce ~33 M tokens per dollar—about 30 % cheaper per token because of six‑fold higher compute and three‑fold higher bandwidth.

Future Directions and Alternatives

Other GPUs (e.g., NVIDIA A10, AMD MI series) may offer better price‑performance than both H100 and 4090. Quantization can shrink model size, allowing inference on a single consumer GPU. Decentralized inference using home‑network‑grade 10‑Gbps links or even blockchain‑based proof‑of‑work with LLM inference is proposed as a way to democratize AI compute while keeping costs low.