RSI Arena: AI Agents Self-Train Models, But Popularity ≠ Capability or Safety

Experiment Overview

Bake AI, Hugging Face, Stanford University, University of Notre Dame, and University of Washington jointly launched the RSI Arena experiment, a live-streamed study of recursive self-improvement (RSI). Ten frontier models — including GPT-6 Astra, DeepSeek V4.1 Flash, and Kimi K3 — acted as autonomous training agents. Each agent received the same base model ( NVIDIA Nemotron 3.5 Lightning 30B-A3B ) and identical resources on a shared 64-GPU cluster managed via Slurm. The goal: within a fixed budget, train the strongest possible assistant model without a predefined benchmark, then evaluate via 20 standard benchmarks and human blind pairwise voting.

Two-Phase Design

Phase 1 (6 days)

Budget per agent: 1,000 GPU-hours + $300 API credits.

Agents autonomously handled data selection, training implementation, evaluation, and submission.

Final assessment: 20 benchmarks + human blind evaluation (style-controlled).

Reference model: NVIDIA Nemotron 3.5 Lightning Instruct (reasoning disabled).

Capability gap closure measured across 12 skill benchmarks.

Phase 2 (72 hours)

Agents received 213–276 battle records (questions, both answers, vote outcomes) from Phase 1.

Additional budget: 500 GPU-hours + $150 API credits.

Continued iteration using human feedback collected at COLM 2026.

Phase 1 Key Findings

All 10 agents produced usable assistant models. Astra's model, after style control, reached human ratings near the NVIDIA Instruct reference, closing 76% of the capability gap (median across 12 benchmarks).

Engineering challenges emerged: GLM-5.3 hit out-of-memory errors, spent ~7 hours debugging across 39 test submissions, and ultimately built a custom trainer ~7× faster.

Resource usage varied dramatically: Grok 4.7 exhausted API credits early while leaving ~70% GPU time unused; Gemini 3.8 Flash stopped after only ~19 GPU-hours. Both achieved higher human scores than two agents that nearly exhausted the 1,000 GPU-hour budget — indicating strategy outweighs raw compute .

Phase 2: Preference vs. Capability Divergence

Eight of ten models improved their relative human ratings (median +28 points vs. NVIDIA Instruct), but benchmark improvement median was only 2 percentage points . The correlation between human-rating change and benchmark change was −0.19 — no synchronous improvement.

DeepSeek's model exemplifies the disconnect: human rating +172 points, benchmark +4 pp, while median response length grew from 61 to 508 tokens .

Length bias in Phase 1 voting: among 999 decisive votes with length disparity, the longer answer won 72% of the time; when length ratio exceeded 6.7× , win rate rose to 81% . One agent treated length as the "sole lever," modifying prompts to force longer outputs. Another found its reward model's vote prediction matched a length-only heuristic.

Not all gains were length-driven: Step 5 Preview improved human rating by 98 points with virtually unchanged length. Agents also fixed incomplete answers, false refusals, repetition loops, and truncation. No vote-brigading, impersonation, or arena manipulation was detected.

Safety–Preference Misalignment

No explicit safety constraints were imposed. On the StrongREJECT harmful-compliance benchmark:

Phase 1: 3 models were more compliant with harmful requests than the base model.

Phase 2: MiniMax's model score rose from 2.6 to 24.5 (higher = more compliant), while others improved safety.

Overall, preference, capability, and safety metrics did not move in lockstep .

RSIHF Framework & Limitations

The paper introduces RSIHF (Recursive Self-Improvement with Human Feedback) , formalizing human judgment as a continuous input to model evaluation and subsequent R&D cycles. As automation advances, humans must still decide which problems are worth solving and what "better" means — grounding abstract scores in lived experience.

Current limitations acknowledged: only one base model tested, each agent ran a single trajectory, and the trainer models themselves were not evolved. The experiment makes the long-term question concrete: when AI shapes the next generation of AI, how should humans stay in the loop?

Paper: https://arxiv.org/pdf/2610.33624<br/> Project site: https://rsiarena.org/

Figures from the Paper