Physical RSI: Can Robots Learn to Learn Better? Framework & Challenges

Paper Overview

Researchers from Shanghai Jiao Tong University, UC Berkeley, Stanford, MIT, Google, Microsoft, NVIDIA, and Xiaomi present the paper "Who Improves the Robot? Toward Physical Recursive Self-Improvement" (arXiv:2610.18842, GitHub: SAIL-Research-Lab/Physical-RSI). The paper introduces the Physical RSI framework, which explores how robots can use physical interaction experience to continuously update both their task execution systems and the improvement processes themselves, aiming for recursive self-evolution.

Core Contributions

A unified framework for Physical RSI with L0–L5 levels, distinguishing whether updates target task capabilities or the improvement process, and whether gains are task-specific or transfer across tasks.

An end-to-end implementation path covering Harness adaptation and model training, analyzing how memory, skills, workflows, and training methods affect task execution and subsequent improvement, integrating safety control and update verification.

Criteria for verifying recursive gains: under identical initial system, experience, and resource budget, compare cross-task evaluation of old vs. new improvement processes to identify evidence gaps in existing work.

Physical RSI Basic Framework

The framework divides the system into two parts: (1) the task execution system interacting with the environment, comprising models and Harness components (prompts, memory, skills, control code); (2) the internal improvement process that updates both the execution system and itself. The main improvement loop starts with a physical interaction, yielding evidence and internal signals. The improvement process proposes modification candidates, a verifier validates them, and an independent evaluator assesses whether the updated system yields better subsequent improvements.

L0–L5 Levels

L0: No autonomous updates; replanning per task or human-dependent improvement.

L1: Autonomous updates to task execution system, gains on same task, fixed improvement process.

L2: Autonomous updates to improvement process itself, gains on same task.

L3: Autonomous updates to task execution system, cross-task gains, fixed improvement process.

L4: Autonomous updates to improvement process, cross-task gains.

L5: On top of L4, independent controlled comparison confirms the new process yields better subsequent cross-task gains than the previous version.

Current Research Progress

Harness Adaptation: From Peripheral Systems to Improvement Method Evolution

System resources outside the model—memory, skills, workflows—can be improved using execution feedback for failure diagnosis, memory updates, skill and workflow updates. Effective modifications are retained; modification records support later diagnosis and strategy selection. Recent work begins adjusting the methods that generate these updates, including diagnostic guidance, memory construction strategies, and search programs, providing references for evolving improvement methods in physical settings.

Model Training: Physical Experience Enters the Training Mechanism

Data collection, model update, and re-execution form a training loop that continuously converts physical interaction data into subsequent improvements. Some works further treat the training mechanism itself as an improvement target; data preparation research adjusts data ratios, simulation distributions, or sampling strategies based on learning outcomes. These explorations extend physical feedback from model parameter updates to data preparation and training method adjustments.

Safety: The Learning Process Must Also Be Safe

A final policy passing safety tests does not guarantee the exploration process was risk-free. Safe reinforcement learning provides constraints during exploration; safe transfer research ensures conditions for reuse. These advances target specific components and scenarios; maintaining safety guarantees while the improvement process itself evolves, and assessing cumulative risk from repeated trials, remain key Physical RSI challenges.

Verification: Does the Modification Work? Are Old Skills Preserved?

Code execution is only the first check. Whether a modification fixes the original failure must be judged by re-execution; changes to shared skills require regression testing on previously successful tasks. Internal verifiers decide which updates are kept; external independent evaluators validate actual gains. If the verifier itself is modified, an external evaluator must judge the new version's effectiveness to prevent the system from gradually relaxing its acceptance criteria.

Benchmark: From Testing Capability to Testing Improvement Methods

Embodied benchmarks like RoboDojo provide task performance and generalization tests; AI4AI-Bench evaluates method or data strategy updates in digital environments. However, the paper notes a lack of joint benchmarks for Physical RSI, especially direct tests of recursive improvement gains.

Challenges and Future Outlook

The paper identifies six open problems:

Learning from uncertain physical experience: How to extract reliable learning signals and prevent erroneous feedback from being amplified through the improvement mechanism.

Causal attribution among Harness components: How to pinpoint the modification target with limited physical trials and disentangle each modification's actual effect.

Non-stationarity of the improvement loop: Whether past effective modifications and improvement strategies remain valid as the system, environment, and existing capabilities change.

Reward hacking and evaluation leakage: How to prevent the system from exploiting loopholes to achieve high scores and maintain evaluation independence so results reflect true physical capability.

Safety preservation during recursive updates: How to maintain safety guarantees and control cumulative risk from multi-round trials as the improvement process changes.

Benchmarking recursive gains: Whether a successor system performs better due to more data, a stronger starting point, or genuine improvement mechanism updates.

Conclusion

Physical RSI extends robot autonomy from task execution to continuous updating of capabilities and improvement methods. The L0–L5 framework maps current progress and highlights a critical gap: task performance improvement alone does not prove the improvement process itself has become more effective. Advancing further requires reliable physical feedback, clear causal attribution of modifications, and safety guarantees that persist through system and environment changes. The ultimate goal is robots that can safely accumulate capabilities and continuously improve their own learning methods when facing new tasks and challenges.