Why Faster Inference Makes Models Smarter: Jonathan Ross Explains GPU‑LPU Synergy

Jonathan Ross, founder of Groq and co‑inventor of Google’s TPU, discusses how inference speed influences the intelligence ceiling of large language models (LLMs). He asserts that faster inference extends a model’s search depth and reflective capability, turning latency reduction into a core intelligence enhancer rather than a mere engineering optimization.

GPU + LPU Complementarity

Ross contrasts Groq’s Language Processing Unit (LPU), launched in 2016, with traditional GPUs. While GPUs excel at compute‑intensive tasks, LPU targets memory‑throughput‑heavy operations such as weight application. He likens the relationship to a logistics network that needs both 18‑wheel trucks and last‑mile delivery vans.

In LLM inference, attention mechanisms are limited by raw compute, whereas weight loading is constrained by memory bandwidth. By assigning compute‑heavy kernels to GPUs and bandwidth‑heavy kernels to LPUs, Groq’s team achieved superior performance across the full performance curve—a strategy originally proposed by COO Sunny Madra and validated through internal benchmarks.

Impact on Model Capability

Ross introduces the claim “the faster the inference, the smarter the model.” Faster hardware enables deeper search in decision trees, akin to the “fast thinking vs. slow thinking” analogy. He cites AlphaGo as a concrete example: on GPUs AlphaGo scored an ELO of ~3200, while on TPUs the score rose above 3900, and the famous “Move 37”—a low‑probability but decisive move—was discovered only when inference speed allowed deeper exploration.

Multi‑Agent Collaboration

Reduced latency also benefits multi‑agent systems. Humans tolerate 1–2 seconds of delay, but for AI agents this is wasteful. LPU’s sub‑second response lets an AI assistant spawn subordinate agents to research tools in parallel, creating a self‑reinforcing workflow.

Leadership Insights

Beyond hardware, Ross shares his perspective on organizational leadership for AI companies. He emphasizes quantifying core objectives, intentional leadership, and transparent information flow. He proposes using “real commercial numbers” and “manufacturing dissatisfaction” as criteria to select talent capable of thriving in long‑term competitive cycles.

Business Context

Groq was founded in 2016, and in December 2025 NVIDIA acquired its IP for $20 billion, completing the deal in three weeks. The integrated technology debuted at NVIDIA GTC 2026 on the Vera Rubin platform, showcasing the practical rollout of the GPU‑LPU hybrid approach.