Why More Compute Can't Fix LLM Inference Lag and Why RL Leads to Overtraining

Reiner Pope, former Google TPU architect, discusses why stacking raw compute power does not eliminate LLM inference latency. He starts by explaining the “fast mode” offering: users pay up to six times more to receive roughly 2.5× higher streaming token generation speed, achieved by reducing the number of concurrent users per GPU.

Low‑concurrency (tiny batch size) means the GPU processes only one or a few requests at a time. In this regime, the physical memory‑bandwidth, not the FLOP capacity, becomes the dominant factor because each generated token requires a full read of the model weights from VRAM. Pope uses the Roofline model to show that total inference latency is bounded by the maximum of compute time (t_compute) and memory time (t_memory). Since per‑token compute is minimal, latency is essentially model‑size‑over‑bandwidth.

He quantifies the economic impact: low‑concurrency reduces hardware efficiency by hundreds to thousands of times. To regain efficiency, the system must increase concurrency until compute and memory costs balance, which he estimates occurs at roughly 300 × model sparsity. This “optimal concurrency” spreads the weight‑read cost across many users.

When concurrency grows, each request’s long‑context KV cache consumes VRAM, creating a disparity between input‑stage (highly parallel matrix ops) and output‑stage (sequential token generation) hardware utilization. This leads to differentiated pricing: longer contexts (e.g., >200 k tokens for Gemini 3.1) incur a 50 % price increase because cache reads become the new bottleneck.

Regarding pipeline parallelism, Pope argues it becomes futile for massive mixture‑of‑experts models such as DeepSeek V3. Although only a subset of parameters is active per token, the total parameter count forces cross‑GPU or cross‑server placement, causing massive inter‑node data transfer during token routing, which outweighs the compute savings.

Finally, he addresses the reinforcement‑learning era: scaling laws like Chinchilla clash with heavy post‑training RL fine‑tuning. Top AI labs continue expensive “over‑training” because RL objectives demand additional data passes, even though this practice is economically inefficient.