Jeff Dean & Fei-Fei Li: RSI's Physical Limits, Self-Improvement vs. Recursive Self-Improvement

Dialogue Overview

At a Stanford HAI event, Stanford professor and World Labs co-founder Fei-Fei Li conversed with former Google Chief Scientist and Discovery Loop co-founder Jeff Dean. The discussion centered on AI self-improvement, recursive self-improvement (RSI), autonomous scientific discovery, mathematical reasoning with formal verification, and pivotal technology milestones.

Discovery Loop: Motivation and Approach

Jeff Dean explained that Discovery Loop emerged from long-standing work on automated systems for software development. Quoc Le's neural architecture search (NAS) research a decade ago represented an early form of recursive self-improvement. Previously, search spaces were defined with rough Python code; today, natural language can instruct a "strong model + agent" system to explore architectures for specific tasks like MRI interpretation. Dean views this as a profound shift in how computer and AI system capabilities are understood, with particular promise for science and engineering.

Self-Improvement vs. Recursive Self-Improvement (RSI)

Dean drew a clear distinction:

Self-improvement : A system optimizes toward a measurable objective (e.g., model accuracy on MRI diagnosis) by trying different architectures and recording results. The underlying proposal mechanism remains unchanged.

Recursive self-improvement : The improvement process itself improves the system that proposes improvements — the meta-optimizer gets better.

Fei-Fei Li emphasized that RSI is often used loosely in Silicon Valley and sought a rigorous definition to separate fiction from fact.

Physical Limits of RSI

Dean argued that RSI cannot accelerate indefinitely. Creating and improving foundation models requires actual training runs and experiments that must physically complete: compute execution, result observation, data saving, and a model deciding the next step. These real physical processes impose a speed limit. Automated systems can increase improvement velocity but not achieve infinite acceleration.

Human Analogy: Cooking and Learning to Learn

Li used an omelette-cooking analogy: burning the first attempt, improving on the second, and reaching an acceptable result after ten tries illustrates self-improvement. Recursive self-improvement would involve the brain's neural connections changing to become a better omelette-maker, perhaps even growing a sixth finger. Dean agreed humans exhibit both: we learn tasks (self-improvement) and learn how to learn (recursive self-improvement), enabling faster acquisition of new skills.

Discovery Loop's Three Focus Areas

Dean outlined Discovery Loop's current concentration on fully digital scientific and engineering loops, which are easiest to accelerate:

Machine learning research

Machine learning engineering

Hardware design

These domains leverage the team's expertise and provide foundational support for other fields. For instance, an automated ML engineering system could be applied to medical or materials science problems to produce specialized models — an example of self-improvement, not necessarily RSI.

AI and the Scientific Method: Experimentalist vs. Theoretical

Li, a physics undergraduate, questioned whether AI agent loops favor an experimentalist philosophy over theoretical physics (like Newton or Einstein). Dean responded that such systems can also pursue theory: they can be asked to propose a theory explaining all data and iterate toward the simplest explanatory framework. Moreover, a system solving millions of problems across a hundred fields would accumulate cross-disciplinary insights unbounded by traditional departmental silos. Dean highlighted AI's "cross-institutional" nature as a catalyst for interdisciplinary communication.

Mathematics, Lean Verification, and the Value of Elegance

Recent AI breakthroughs in mathematics (Navier-Stokes, 720 papers, OpenAI results) prompted mixed feelings from Li. Terence Tao's "Math 2.0" talk noted AI solves some problem classes but not those lacking training data. Dean acknowledged models and highly parallel agent systems can now resolve extremely hard unsolved problems, revealing current capability boundaries. However, proofs often appear as 500-page Lean formalized code — machine-verified but humanly opaque. Dean stressed: "Knowing a problem is proven or disproven has value, but understanding how to develop elegant techniques to reach the result is equally important." Elegant solutions yield transferable tricks; lengthy machine proofs may not. The fact that many problems are now solvable is reality; the community must decide how to leverage this for deeper mathematical research.

Key Technology Milestones and Two Surprises

Dean identified three pivotal technologies:

Transformer : Efficient architecture preserving all prior states via attention, unlike LSTMs that compress history into a small state vector. Enables learning more with less compute.

Distillation : 2016 NeurIPS-rejected paper (with Oriol Vinyals and Geoff Hinton) that became crucial for efficient inference — training a massive teacher model to guide a lightweight student.

Mixture of Experts (MoE) : Proven successful, but Dean expected more "exotic, organic" architectures (multiple neural columns with sparse long-range connections). Instead, MoE remains regular: uniform expert sizes, quick re-aggregation. NAS-generated models show more organic connectivity than hand-designed ones.

Two unexpected developments:

MoE did not evolve toward the wild, organic architectures Dean anticipated.

Code capability importance far exceeded expectations — coding became the key "unlocker" for model potential across diverse problems. This was a focus in Gemini but not the top priority at the time.

Societal Impact: Amplifying Benefits, Mitigating Harms

Dean co-authored "Shaping AI's Impact on Billions of People" (shapingai.com), examining seven domains (employment, AI for Science, misinformation, media, etc.) with domain experts. His primary concern: maximizing positive gains across all affected fields while minimizing negative consequences. This requires not just technical solutions but regulation, policy, and citizen participation — e.g., providing adaptation pathways for workers displaced by AI.

Advice to Students

Dean advised understanding emerging technologies and combining them with one's genuine passions — whether medicine, writing, or software engineering. Human agency remains central: people decide what we collectively pursue. Li concluded: "In the AI era, human passion still holds timeless value."