Inference Foundry: Token Physical Cost and Exploding Demand Force Heterogeneous Division

Token as the Atomic Unit

Token is the smallest tradable unit of AI, analogous to a transistor in semiconductors or a kilowatt‑hour in electricity. Generating a token incurs an irreducible physical cost: GPU compute time, HBM bandwidth, and electricity, all scaling linearly with token count.

Demand Explosion

Inference‑time reasoning : modern models iterate, verify, and backtrack, raising per‑query token consumption from a few hundred to tens of thousands.

Agents : AI agents execute tasks across email, file systems, and approval flows, consuming hundreds of thousands to millions of tokens per session.

When AI capabilities surpass human ability, willingness to pay shifts from a gradual curve to an explosive, discontinuous jump.

The Cube: Simultaneous Growth Axes

Demand : exponential growth in concurrent queries.

Model size : per‑query compute expands with parameter count, context length, and output length.

Innovation speed : new models and capabilities continually raise the ceiling of the other two axes.

Even with a 50× hardware improvement every five years, the combined multipliers (≈14× for parameters, ≈30× for context, ≈250× for output tokens) outpace hardware gains, keeping total inference CapEx on the rise.

Physical Cost of a Token

Per‑token FLOPs exceed those of a database query, video encode, or physical simulation. The total compute per token is driven by three variables: model parameter count, context length, and output token count.

FFN layer (compute‑bound) : each token passes through a full‑width feed‑forward network. For a 1 T‑parameter dense model, per‑token forward FLOPs are ~2 T, dominated by the FFN.

Attention layer (bandwidth‑bound) : each token reads the KV‑cache for the entire context. The bottleneck is HBM bandwidth, not raw compute.

KV‑cache compression (GQA/MQA → MLA → DeepSeek V4 CSA+HCA) reduced a 1 M‑token cache from hundreds of GB to ~9.6 GB, yet each decode step still reads the full cache.

Current GPU HBM bandwidth limits (B200 ≈ 8 TB/s, B300 ≈ 16 TB/s) set a lower bound on per‑request latency, explaining why Groq’s LPU with 150 TB/s on‑chip SRAM bandwidth offers real value for the decode phase.

Optimization vs. Demand

Mixture‑of‑Experts (MoE) and compressed attention shrink the compute‑bound and bandwidth‑bound components respectively, but they do not curb the exponential rise in concurrent queries, which is a pure demand‑side phenomenon. The product of per‑query compute inflation and concurrent query growth constitutes Sunny’s “cube”.

Cube Breakdown (2.1)

Demand: exponential increase in concurrent queries.

Model size: per‑query compute inflation (14× for parameters, 30× for context, 250× for output tokens).

Innovation speed: continual model and capability upgrades.

Combined, these multipliers yield a ≈10 000× growth in total compute demand.

NVIDIA’s Acquisition of Groq

Groq, founded by Google TPU inventor Jonathan Ross, builds a purpose‑built inference architecture (LPU) rather than repurposing training GPUs.

Architectural contrasts :

Scheduling model : GPUs use dynamic hardware schedulers; LPUs employ a deterministic data‑flow compiler, eliminating runtime branch prediction at the cost of flexibility.

Memory hierarchy : GPUs rely on off‑chip HBM; LPUs use on‑chip SRAM with 150 TB/s per chip, enabling far higher bandwidth for decode‑stage accesses.

Design philosophy : GPUs aim for generality (training + inference, large + small models); LPUs specialize in inference‑only, bandwidth‑intensive decode.

NVIDIA’s NVLink Fusion protocol lets third‑party accelerators plug into the NVIDIA fabric, allowing the Attention stage to stay on GPU (HBM) while FFN/MoE runs on the LPU (SRAM). In Sunny’s demo, the heterogeneous solution produced ~2.5× more tokens than a same‑generation GPU‑only system under equal power budget.

Disaggregation Layers

First layer : pre‑fill (compute‑bound) vs. decode (bandwidth‑bound) split, now standard in serving stacks such as Dynamo, vLLM, SGLang, and MoonCake.

Second layer : internal decode split—Attention stays on GPU, FFN/MoE moves to LPU (Attention‑FFN Disaggregation, AFD). The combined system, linked via NVLink, achieves 2–3× same‑generation efficiency, matching the 2.5× demo figure.

First‑layer disaggregation already yields 2–7× throughput gains in production deployments.

Industry Implications

Inference is evolving from a single compute task into a multi‑stage production line, mirroring semiconductor manufacturing’s move from monolithic fabs to specialized lithography, etch, and packaging steps.

The ultimate bottleneck is power. A GW‑scale data center requires ~10 GW of electricity—far exceeding typical national grid upgrade timelines (7–10 years). Consequently, the industry pursues three parallel power strategies:

Natural‑gas‑behind‑the‑meter modular data centers (fastest to deploy, limited to <200 MW per site).

Nuclear SMRs and re‑started legacy plants (zero‑carbon, GW‑scale, but long commercialization timelines).

Renewables plus storage (lowest LCOE but intermittent, requiring 4–8 h batteries or hydrogen).

Securing long‑term offtake contracts (take‑or‑pay style) becomes as critical as securing GPU capacity.

Inference Foundry Concept

An inference foundry must simultaneously satisfy three conditions:

Self‑designed chips to escape NVIDIA’s pricing power.

Owned power infrastructure to overcome the ultimate energy constraint.

Long‑term offtake agreements to guarantee utilization without competing on the model layer.

Current players fall short: CoreWeave relies on NVIDIA GPUs and fragmented power; Oracle’s Stargate has ambitious targets but limited deployed capacity; Groq’s prototype demonstrates feasibility but still depends on NVIDIA fabric.

The sector’s value chain will shift from “who has the best model” to “who can turn electricity into tokens most efficiently.”

Conclusion

Tokens carry an irreducible physical cost; demand grows exponentially; a single architecture cannot saturate hardware. Heterogeneous division—culminating in an inference foundry—is inevitable.

References

Stanford MS&E 435: Economics of the AI Supercycle — Class #2, Guest Speakers: Brad Gerstner (Altimeter) & Sunny Madra (Groq / NVIDIA). Spring 2026. https://www.youtube.com/watch?v=4faCRNl9Bi4

Agrawal, Apoorv. “The Economics of Generative AI: Two Years Later.” Tailwinds (Substack), April 1 2026. https://apoorv03.com/p/the-economics-of-generative-ai-two

CNBC. “Anthropic set to hit $10.9 billion in revenue during second quarter, source says.” May 20 2026. https://www.cnbc.com/2026/05/20/anthropic-revenue-explosive-growth-ipo-profitable-quarter.html

The Information. “OpenAI’s AI Chip Deal With Broadcom Hits $18 Billion Financing Snag.” May 2026. https://www.theinformation.com/articles/openais-ai-chip-deal-broadcom-hits-18-billion-financing-snag

Tom’s Hardware. “Google, Microsoft, Meta, and Amazon capex spending to hit $725 billion in 2026, up 77% from last year.” April 2026. https://www.tomshardware.com/tech-industry/big-tech/big-techs-ai-spending-plans-reach-725-billion