RSI in AI: Definitions, Technical Paths, Bottlenecks, Benchmarks & Human Roles

Introduction: The RSI Discussion Context

The transcript captures a 2-hour AMA (青稞AMA 第4期) hosted by He Bingxiang (PhD student, THUNLP) with five guests: Ma Xiaoteng (co-founder, Mind Lab), Huang Chengsong (PhD, WashU), Qian Cheng (PhD, UIUC), Yang Junlin (PhD, Tsinghua), and Zhang Kaiyan (CTO, Xianyuan Tech). The discussion opens with industry signals: Anthropic reports >80% of merged code comes from Claude (since May 2024), and OpenAI's Research Agent runs 3.1x the hours of human researchers (as of Aug 2024), indicating AI already handles substantial coding and experimentation work.

Defining RSI: Recursive vs. Iterative, Self vs. AI-for-AI

Zhang Kaiyan: RSI Requires System-Level Self-Improvement

Zhang distinguishes RSI from AI4AI. RSI's "R" (Recursive) and "S" (Self) imply the entire AI system—model, harness, data pipeline, training/inference infrastructure—improves itself in a closed loop: the improved model then further optimizes the system. He cites Anthropic's blog diagram: Claude 3.5 → Claude 4 → Claude Code, which then optimizes training data synthesis, inference, etc., creating a self-reinforcing cycle. By contrast, most current work uses a stronger AI to build a specialized AI (AI4AI), not self-improvement of the same system.

Huang Chengsong: Data as the Medium for Recursive Self-Improvement

Huang defines RSI as the model generating data that enhances its own capabilities, then using that improved capability to generate better data, continuing the loop. The key is minimizing human-in-the-loop. His work (Rzero, Environment Harness) focuses on text reasoning models and agent environments where the model reads/processes data to self-improve. He notes algorithm and data are orthogonal; data remains essential regardless of algorithm changes (pre-training → RLHF → RLVR → Group-based/Critic-free post-training).

Yang Junlin: Pragmatic Measurement via AI Participation Ratio

Yang references OpenAI's technical report metric: percentage of R&D effort contributed by GPT in developing the next generation. This tracks progress toward zero human involvement. He notes the ratio has risen each generation, offering a concrete, if imperfect, gauge.

Qian Cheng: Component-Level Definition and System Boundaries

Qian asks whether AI4AI with the same model for both roles counts as RSI. He emphasizes defining the optimization target: model weights, harness (tools, memory), training algorithms/infrastructure, or task environments/curricula. System boundaries must be clarified before a unified definition emerges.

Ma Xiaoteng: Industrial View — Reward Generation and Data Challenges

Ma highlights two shifts: (1) reward/environment generation moving from human RLHF/verifiers to stronger models acting as judges (e.g., GLM, DeepSeek V4, Kimi now achieve high alignment vs. Qwen3-30B's 60-70% in Jan 2024); (2) data acquisition remains the core industrial bottleneck — AI cannot autonomously gather new data; companies like Anthropic acquire firms for data ("Money for AI"), and noisy user feedback requires human-aligned evaluation, which models still struggle with.

Technical Routes for RSI / AI4AI

Qian Cheng's Four-Category Taxonomy

Policy layer : data and model weight improvement.

Harness layer : external components (tools, memory, routers, verifiers) — Qian's focus; test-time optimization here.

Training algorithm layer : model architecture, infrastructure, training algorithms.

Task layer : evaluation environments, curricula, benchmark evolution.

He notes harness optimization can boost performance without new policy parameters, and the paradigm can transfer to task environments (evolving benchmarks via failure diagnosis).

Huang Chengsong: Data-Centric RSI as a Durable Investment

Huang argues harness optimization is lightweight and suits personalized/domain-specific agents, but fundamental model improvement requires data. Data-centric RSI contributions persist even if algorithms change. He contrasts harness-only RSI (limited by model's existing capability, no new information gain) with data-level RSI (new data brings larger gains).

Yang Junlin: Frontis-MA1 — Large-Scale AI4AI Gyms and Evolutionary Search

Yang describes OpenMLE-Gym : 5,000+ synthesized ML tasks (vs. hundreds in prior open-source) built from internet ML tasks, enabling SFT/RL training of AI4AI capabilities. Also NatureBench from Nature Machine Intelligence papers for AI-for-Science. On search, he notes naive iteration loses information gain after ~20 rounds; they use OpenMLE-Evo search trees (Monte Carlo tree style) to manage exploration, selecting high-value parent nodes. He highlights the open challenge: can the LLM itself decide the evolutionary search strategy (parent selection, exploration direction, backtracking)? Hyperagents attempted this but haven't surpassed human-designed search yet.

Ma Xiaoteng: Infrastructure & Model Iteration Speed via LoRA

Ma stresses iteration speed: if you train once a year, search can't trial enough. Better infra (e.g., DeepSeek V4.1 architecture boosting throughput) directly raises success rates. His team uses LoRA for lightweight adaptation: dozens of LoRAs on a base model, hot-swappable, enabling rapid version rollback and continual learning. He aligns with Huang: seek invariants (data, infra efficiency) amid algorithmic change.

Ma on RSI Infrastructure: Reef as a Self-Iteration Harness

Ma endorses Reef as a system managing versioned training data, hyperparameters, experience collection, web data sourcing, and staged rollouts (5% traffic for new checkpoints). Combined with multi-LoRA switching, it provides the version-control and deployment backbone for AI self-iteration.

Core Technical Bottlenecks

Qian Cheng: Failure Diagnosis — Bottom-Up vs. Top-Down

Current coding agents diagnose bottom-up from massive case logs, lacking the top-down macro perspective humans use (e.g., reading loss curves to pinpoint issues). This leads to overfitting or misdiagnosis, limiting self-improvement efficiency. Reducing human oversight requires breakthroughs here.

Zhang Kaiyan: Environment Construction & Search Efficiency

Two interaction bottlenecks: (1) Environment feedback — digital data largely exhausted; physical-world environments (wet labs, long-cycle experiments) are slow and expensive (Anthropic acquires companies for envs). World Models from embodied AI may accelerate feedback. (2) Search/novelty — generating truly novel solutions and evaluating novelty remain hard; without good novelty rewards, search fails.

Yang Junlin: Novelty Definition & Credit Assignment in Long Horizons

(1) Novelty judgment currently requires human review of trajectories; automating this via LLM judge/verifier is unsolved. (2) Credit assignment — innovative paths involve many failed steps; penalizing all failures prevents reaching success. How to reward exploratory failures that lead to breakthroughs?

Ma Xiaoteng: Data Production Speed & Long-Term Cognitive Memory

(1) Heavy tasks (e.g., TB4.0 debugging takes days) yield few samples per day — a fundamental throughput bottleneck; World Model mock simulation may help. (2) Long-term memory — models lack persistent cognitive storage. Humans internalize hundreds of papers into intuition; models must stuff everything into context. Ma suspects OpenAI's week-long reasoning uses architectural changes (beyond harness) to inherit accumulated "sense" across steps. He explores TTT (Test-Time Training) and long-term memory models.

He Bingxiang (Host) adds: Noisy evaluation & premature convergence

In PostTrainBench experiments, a 2-point benchmark dip caused the agent to permanently discard a potentially correct action direction (misattributing noise or implementation error), akin to a bandit algorithm abandoning an arm after one negative reward. Agents are overly conservative and misattribute causality.

Benchmark Landscape & Evaluation Philosophy

Yang Junlin: Categorizing RSI Benchmarks

AI4AI benchmarks repackaged as RSI — valuable because AI4AI capabilities transfer to RSI.

Post-training benchmarks : full-stack (PostTrainBench, 10-hour runs) vs. sub-component (RSI-Bench-Data). Decomposing post-training enables faster iteration (e.g., AutoSFT optimizing SFT alone).

Ma Xiaoteng: Practical Use & Harness Co-Evolution

His team runs PostTrainBench and RSI-Bench. They note the benchmark's default Tinker harness can be replaced with their own optimized harness, creating a "nested" improvement loop: better harness → more trials on RSI-Data tasks → better results. Main complaint: benchmarks are slow.

Qian Cheng: Need for Recursive, Long-Horizon Benchmarks

Current benchmarks (fixed budget, single-round optimization) miss the recursive essence: checkpoint from round 1 must feed into round 2, measuring sustained improvement slope over multiple cycles, whether at policy, harness, environment, or data level.

Zhang Kaiyan: Cost/ROI & Multi-Objective Real-World Metrics

Industry cares about investment-output ratio (lower cost for better model) and multi-objective Pareto fronts (performance, inference speed, deployment footprint). Current SOTA benchmarks ignore these; they need real user-defined multi-dimensional targets. Goodhart's Law applies: once a metric is targeted, it ceases to be a good metric.

Huang Chengsong: Domain-Specific & Subset-Based Benchmarks

General LLM benchmarks miss domain-specific needs (requiring domain experts). High run-cost (time/money) hinders optimization. Ideal: a few representative tasks per domain across hundreds of domains, with small subsets predictive of full performance.

He Bingxiang: Modular Diagnostics — AutoRLBench

End-to-end benchmarks don't guide how to improve. The loop requires: diagnose run health → locate root cause → propose better action. They are building AutoRLBench for RL diagnosis/repair, calling for community contributions.

Future: What Counts as RSI, Timelines, Human Roles

Yang Junlin: RSI = AI Does Most R&D Humans Set Direction

Intuitive definition: GPT does most/all research for next GPT; humans provide high-level goals/metrics.

Huang Chengsong: RSI System Outputs Complete Agents; Humans Become PMs (~3+ years)

RSI realized when we specify desired agent function and system outputs the full agent (data→training→harness). Requires very strong base models. Humans shift from engineers to product managers defining "what", not "how".

Qian Cheng: Automated Research ≠ RSI; 2028 Target for Automated Researcher; Humans Drive Cross-Domain Ideas

OpenAI targets 2028 for fully automated AI researcher. True RSI (continuous recursive improvement) may take longer. Humans should focus on applying AI tools to basic sciences (AI for Science), bridging the gap between what scientists need and what agents provide. Human value: creative ideas/topics for agents to execute.

Ma Xiaoteng: RSI as Spectrum, Not Binary; 1-Year Horizon for Commercial Training Pipeline; Two Societal Futures

RSI is continuous (like AGI/Emerge concepts). OpenAI/Anthropic already operate at 90% AI R&D effort — effectively RSI. For his commercial training delivery, a 1-year target: align benchmark/data with client (human FDE), then agent handles training/continual learning. Two futures: (A) super-AI causes mass unemployment/social collapse; (B) tools enable everyone to train own models, own benchmarks, offer services — a "colorful" society where individuals become "Hermès of their domain" via unique taste/data pipelines. Researchers work to realize B.

Zhang Kaiyan: RSI as Continuous Optimization Spectrum; AI Replaces 99% Mental Labor; Humans Audit & Verify

RSI is a spectrum (Claude Code already optimizes 80-90% internal code). For frontier labs, it's an optimization problem (scaling curves). For others, AI creates tools that replace mental labor (1-3% now → 99% later), potentially triggering scientific explosion. Human role shifts to evaluation & verification — critical because unconstrained goal-seeking AI may produce harmful hacks (e.g., GPT in cyberattack). Optimistic long-term: like industrial revolution brought weekends/welfare, mental labor automation may bring 3-day work weeks.

Audience Q&A Highlights

Gating external knowledge (Qian): Purely internal signals hit ceilings (unsupervised RL limit). External knowledge must be injected at the meta level (harness design, training config, data selection), not just low-level execution traces.

Harness optimization ceiling (Huang): Harness-only gains are far below joint model+harness evolution. Harness opt is a resource-constrained trade-off (model weight updates need massive GPU memory).

DreamRSI Meta-Policy vs. Policy (Yang): Policy = single-step optimization (rewrite MLE, write kernel). Meta-Policy = search strategy (which node to expand, parallel vs. serial). DreamRSI lets agent modify its own meta-policy; long-term both should be LLM-driven.

Internal early-warning metrics (Zhang): Full trajectory monitoring (Langfuse, W&B) — token usage, latency, reward, entropy — plus agent-vs-analyzer adversarial setups to catch reward hacking.

Cross-domain MLE scarcity (Ma): LoRA capacity limits; they use Mixture of LoRA (separate checkpoint tracks per domain, switch at inference). General capability emerges only when diverse experiences are merged back into continued pre-training (CPT).

DSH as harness baseline (Qian): Structured, interface-based harnesses (MCP-like) improve generalization and testability; preferred for research.

Will harness disappear? (Qian & Huang): Not in ~3 years. Base models aren't strong enough to solve all tasks without in-context learning / small fine-tunes (harness role). Harness capabilities gradually distill into model; only when base model adapts effortlessly to any domain will harness become unnecessary.