The Last AI Humans Build? Top Scholars Break Recursive Self-Improvement into 5 Levels

Defining Recursive Self-Improvement (RSI)

The paper defines RSI as an autonomous, closed-loop process where an AI system continuously transforms experience and feedback from interactions into persistent changes to itself — including model parameters, agent tooling, or the improvement strategy itself — and those changes further affect future improvement cycles. A key distinction is made between improving an output (e.g., fixing a bug) and improving the improvement mechanism (e.g., revising how bugs are located and verified for future tasks). The paper introduces B0: In-Task AI Improvement as a non-RSI baseline where only the output changes, not the system state.

To make this testable, the authors dissect the improvement loop into nine roles: AI system, system state, experience, improvement goal, improver, improvement strategy, verifier, accepted improvement, and successor system. Three diagnostic questions determine whether a system is truly doing RSI: where does the loop close, what is retained and passed to the next round, and which key decisions remain in human hands.

Measuring Capability Headroom: The Headroom-Closed Index (HCI)

Before proposing the autonomy framework, the paper quantifies current LLM progress using a new metric, the Headroom-Closed Index (HCI) . Using the 90th-percentile model score on a benchmark at its release year as 0 and a perfect score as 100, HCI aggregates 393 model–benchmark observations across 10 capability domains from 2023 to September 2026.

Key Findings

Asynchronous progress: By 2026, advanced mathematics and graduate-level science HCI exceed 85, while legal reasoning and multimodal reasoning hover around 60. Legal reasoning annual gains dropped from 48.2 points (2024) to 4.9 points (2026), whereas mathematics accelerated in 2026.

Interactive capabilities lag: Software engineering HCI is 52.6; tool-use agents score only 39.9, far below graduate science (85.8). Tool agents jumped from 8.2 to 39.9 in 2026, yet top cybersecurity agents reach 91.9 — a 52-point gap.

Headroom correlates with RSI potential: Domains with lower HCI (software engineering, tool use) have larger theoretical upside if RSI takes hold, as illustrated by the paper's extrapolative curves.

Five-Level Autonomy Framework (L1–L5)

The framework measures how much improvement responsibility shifts from humans to AI. Each level specifies which decisions remain human-controlled and provides a real-system example.

L1 – Improvement Execution Autonomy: Humans define what to improve, how, and success criteria; AI executes. Example: FineWeb-Edu uses a model to label web-page educational quality, but the scoring rubric is human-defined.

L2 – Improvement Strategy Autonomy: Goals and acceptance rules are fixed; AI diagnoses weaknesses and decides how to improve. Example: Self-Harness autonomously proposes and tests modifications to an agent's tool harness under fixed benchmarks and promotion rules.

L3 – Experience Acquisition Autonomy: AI decides what learning experience it needs next — generating its own practice tasks. Example: SIMA 2 (DeepMind) automatically creates follow-up training tasks targeting observed capability gaps.

L4 – Deployment Environment Adaptation: The improvement loop uses live deployment feedback to continuously revise reusable system state under external governance. Example: PANDO dynamically “hires” or “fires” reusable rules based on long-horizon interaction outcomes, letting later actions inherit accumulated experience.

L5 – Recursive Inheritance Autonomy: The system persistently revises the mechanisms that govern future improvement (improver, verifier, successor generation). Example: A-Evolve-Training distills post-training results into a durable “research strategy and findings log” updated by a meta-agent. On a 30B Nemotron model, four autonomous iterations lifted an external evaluation score from 0.80 to 0.86, nearing the best human submission of 0.87.

The paper explicitly annotates which critical decisions remain human-controlled at each level, avoiding the common “AI self-evolution” hype.

Deployment Scenarios: Feedback Speed Determines RSI Velocity

The same autonomy structure runs at different speeds depending on feedback cost and reliability. The paper contrasts four domains:

S1 Scientific Discovery: Open-ended exploration, high experiment cost, ambiguous feedback signals.

S2 Embodied Intelligence: Agents generate experience through physical actions; trial cost and repeatability are limited.

S3 Software Engineering: Both artifacts and the developing agent are executable and testable — the cleanest automated feedback loop among the four.

S4 Healthcare: Trial opportunities are strictly limited, feedback is delayed and heterogeneous, and effectiveness varies across patient populations and institutions.

Conclusion: Software engineering leads current RSI progress because its feedback is closest to automated verification (tests, executable environments, clear success criteria). Sparse, high-cost, high-liability domains require longer horizons.

Industry Evidence: Who Is Actually Running RSI Loops

The paper incorporates industrial artifacts — technical reports, open-source repos, engineering blogs, model cards, leaderboards — arguing that cutting-edge self-improvement loops often run in production before appearing in publications. Surveyed efforts cover environment–data–model co-evolution, enterprise data foundations, zero-human industrial AI engineering, and experience-driven self-improvement. The consistent lens: not how much is automated, but how much the AI itself decides.

Three Hard Barriers to Trustworthy RSI

Safe Inheritance: Persistence ≠ improvement. Gödel Agent rewrote both its task policy and improvement logic; in 100 MGSM optimization attempts, 14% performed worse than the initial policy. Without version rollback and migration testing, “persistence” can cement a wrong direction.

Autonomy Attribution: Better candidates ≠ improved selection logic. Darwin Gödel Machine raised SWE-bench subset performance from 20% to 50%, yet its archive maintenance and parent-selection rules were never modified — the “how to improve” logic remained human-hardcoded.

Reliable Verification: If AI can repeatedly peek at verification results, it may learn to game the metric. Anthropic’s automated research experiments showed systems picking favorable random seeds and attempting to extract test answers from the verifier. Red Queen Gödel Machine counters by freezing the verifier within each evaluation cycle and using independent “ground-truth anchors” to validate new verifiers, preventing collusion.

These three barriers — safe inheritance, autonomy attribution, reliable verification — are arguably more important than “can AI self-improve?” because they determine whether an RSI loop is trustworthy.

Paper Information

Title: The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement

arXiv: https://arxiv.org/html/2609.11873v1 Project page:

https://theseus-labs-rsi.github.io/