Recursive Self‑Optimization: Solving AI Infra’s Speed‑Scale‑Complexity Triangle

In the fifth installment of Stanford’s MS&E 435 series, the author examines why AI infrastructure must overcome a “speed‑scale‑complexity” triangle: three‑fold annual compute growth, gigawatt‑level capacity, and heterogeneous DAG orchestration. Traditional approaches can satisfy at most two of these constraints, whereas recursive self‑optimization is presented as the sole method that can address all three.

Key Insights

Compute demand is self‑driving and never stops. Scaling laws apply across the entire AI lifecycle—pre‑training, reinforcement‑learning‑after‑training, synthetic‑data generation, and product services—causing demand to increase generationally without a foreseeable endpoint.

Compute translates directly to revenue, but build‑out speed lags demand by two orders of magnitude. Over the past three years compute capacity has tripled annually, matching revenue growth. However, chip design cycles (~3 years) and data‑center construction (5‑7 years) are far slower than model iteration cycles (months).

Agents turn the computation graph from a single node into a heterogeneous DAG. Inference is only one node type; tool calls, searches, and VM environments introduce new CPU‑bound workloads. GPUs handle heavy inference, specialized accelerators handle low‑latency inference, and CPUs manage orchestration, making heterogeneity inevitable.

Recursive self‑optimization is the only viable solution. Human engineers cannot keep up with the “understand‑design‑deliver” loop given the three‑year chip design horizon versus month‑scale model iteration. Allowing models to define the next generation of chips and software compresses this loop.

Physical‑world scaling laws are orders of magnitude slower than digital‑world scaling. While recursive methods accelerate chip design, software orchestration, and model iteration, the manufacturing of chips, power plants, and data‑center infrastructure cannot be compressed, widening the gap between digital demand and physical supply.

Compute Growth vs. Chip Production

OpenAI’s CFO reported that compute capacity and revenue have grown in lockstep over the past three years, with a three‑fold annual increase. A rough estimate places a 1 GW data‑center’s full‑stack investment at $60‑70 billion, including roughly 500 000 GPU cards. The industry’s ambition—30 GW for OpenAI alone and >100 GW across U.S. hyperscalers—represents more than one‑fifth of the U.S. power grid.

Because the physical world cannot iterate as quickly as the digital world, each new model generation demands more compute (larger context windows, richer reasoning, more tool calls) while the hardware to satisfy those demands arrives years later.

Inference Becomes a Directed Acyclic Graph

Contrary to the common misconception that AI infrastructure equals model training, inference now dominates compute usage and is split into three distinct workloads:

Reinforcement‑learning‑after‑training: Large‑scale rollout generation and verification consume far more inference compute than parameter updates.

Synthetic data generation: Models generate high‑quality data for the next generation, adding a substantial inference load.

Product services: End‑user requests such as ChatGPT conversations, Codex coding, API calls, and enterprise AI applications.

These workloads form a DAG where nodes include inference, tool calls, database queries, VM launches, and result validation. CPUs are resurging because they handle the non‑inference nodes, and heterogeneous accelerators are required to match each node type.

Design Bandwidth: Linear vs. Exponential

The author quotes Sachin Katti: “We have succeeded from a compute perspective when the human becomes the bottleneck.” The current AI infrastructure bottleneck is not GPU supply alone but the exponential growth of the problem space versus the linear design bandwidth of human engineers.

Key constraints include:

Multi‑layer memory architectures beyond HBM for long‑context caching.

Heterogeneous compute units tailored to different DAG node types.

An intelligent orchestration layer that decides which hardware runs each DAG node.

CPU architecture evolved over 40 years through layered caches, DRAM, flash, and distributed storage; AI systems must achieve comparable complexity in 3‑5 years, a timeline that far exceeds human design capacity.

Recursive Self‑Optimization in Practice

Chip Layer: From Years to Hours

Traditional chip design takes three years and hundreds of engineers. NVIDIA’s NB‑Cell uses reinforcement learning to compress a standard‑cell library design from a ten‑month, eight‑person effort to a single GPU run overnight. Google’s AlphaChip applies reinforcement learning to floor‑planning, enabling recursive improvement across TPU generations.

Software Layer: AI Writes AI Software

Anthropic disclosed that by May 2026 Claude contributed over 80 % of the company’s merged code, with engineer productivity eight times higher than in 2024. OpenAI uses Codex to develop and optimize its infrastructure software, creating a closed‑loop where AI builds the software that runs the AI.

Model Layer: AI Creates AI

OpenAI’s roadmap notes that GPT‑5.3‑Codex will be the first model to play a critical role in its own architecture search, training code, data pipeline, and evaluation framework. Anthropic’s Claude Opus 4.5 follows a similar trajectory. Google remains the only company with a long‑standing co‑design partnership between TPU teams and DeepMind.

These three recursive loops form a flywheel: software cycles in days, model cycles in weeks to months, and chip manufacturing still requires 12‑18 months. The acceleration applies only to design, not to physical fabrication.

Design vs. Manufacturing

OpenAI’s “Stargate” project targets a $500 billion total investment, aiming for 10 GW in the near term and 30 GW long term. A 1 GW stack costs $60‑70 billion and includes roughly 500 000 GPUs. Physical constraints—energy, transformer, battery, cooling, and power‑grid capacity—become the ultimate limiting factors.

The author highlights that the most compelling investment opportunities now lie in large‑scale transformers, batteries, power generation, and cooling infrastructure, which have both technical and scale barriers that make them hard to replicate.

Strategic Outlook

Short‑term: Focus on orchestration software, as better scheduling can multiply effective compute without new hardware.

Mid‑term: Develop multi‑layer memory architectures to support long‑context workloads.

Long‑term: Invest in physical infrastructure—transformers, batteries, cooling, and power generation—because these assets have dual technical‑scale barriers.

While recursive self‑optimization can compress design cycles, the physical world’s scarcity will increasingly dominate AI infrastructure performance as digital recursion accelerates.

References

Sachin Katti, “Infrastructure, Capstone Case,” Stanford MS&E 435: Economics of the AI Supercycle, Spring 2026.

Sarah Fryer, “A Business That Scales with the Value of Intelligence,” OpenAI Blog, January 2026.

Bill Dally, “NB‑Cell: Reinforcement Learning for Standard Cell Design,” NVIDIA GTC, March 2026.

Mirhoseini, A. et al., “AlphaChip: AI for Chip Floorplanning,” Google DeepMind, 2023.

Anthropic, “When AI Builds Itself,” Anthropic Blog, June 4, 2026.

OpenAI, “Path to AGI,” as reported by Bloomberg, 2024.

OpenAI & SoftBank, “The Stargate Project,” Joint Announcement, January 2025.