Who Improves the Robot? A Framework for Physical Recursive Self-Improvement

Introduction

Robot execution autonomy and improvement autonomy are distinct. A robot may autonomously perform tasks like manipulation or navigation, yet still rely on humans for failure diagnosis, data collection, demonstration of corrections, training pipeline design, and version selection. Improved execution performance does not imply autonomy in these behind-the-scenes steps.

Ordinary self-improvement changes what is used during task execution (e.g., adding skills, modifying prompts, updating policy parameters). Recursive self-improvement goes further: it changes how future changes are generated, tested, and selected. Only when a new improvement procedure produces more effective subsequent improvements under controlled conditions does it constitute evidence of recursive gain.

The physical world makes this especially hard: robots receive partial, noisy observations; must act under continuous dynamics and real-time constraints; trials, resets, and repeated evaluations are expensive; hardware can wear or break; some consequences are irreversible. Low-cost trial-and-error in software does not transfer directly to robots sharing space with humans.

The authors therefore propose a dedicated Physical Recursive Self-Improvement (Physical RSI) framework: the robot uses physical interaction evidence to continuously update both its task execution system and its improvement procedure, making later improvements more effective. The contribution is not to claim this goal is already achieved, but to provide a leveled standard, organize existing methods, clarify safety constraints, and expose evidence gaps.

Foundations of Physical Recursive Self-Improvement

2.1 Definitions and Scope

The framework first defines four system objects:

Task Execution System : Comprises the out-of-model harness (prompts, memory, skill library, tools, task decomposition, recovery logic, interfaces, controller code) and parameterized models (planners, vision-language-action policies, value functions, world models used at execution time).

Improvement Procedure : Contains three phases — harness adaptation, model training, and internal verification — responsible for modifying the harness, selecting experience and optimizing models, and judging whether candidate changes are retained. It can also modify its own phases.

Experience and Feedback Library : Records execution trajectories and improvement history; distinct from the memory used at execution time, which is a distilled, callable resource extracted from experience.

Independent Evaluator : Uses test tasks and scoring criteria that the improvement procedure cannot read, train on, or modify. It evaluates both the updated execution system and modifications to the internal verifier. Without this, a verifier could relax its own standards and "prove" improvement while actual robot capability stagnates.

Physical self-improvement requires: (1) the system autonomously generates and retains at least one capability update; (2) the retained change has physical grounding; (3) the final capability outperforms the initial version under independent evaluation. Physical recursive self-improvement adds: autonomous retention of improvement procedure updates, and demonstration of cross-task recursive gain — from the same system version, same experience, and same budget, the new procedure produces an execution system that outperforms the one produced by the previous procedure .

"Retention" means the update persists beyond the current episode and becomes the starting point for subsequent versions. "Physical grounding" can come from real-robot or physics-simulation action outcomes; reliance solely on learned world-model imagination, language judgment, or internal signals is insufficient. Physics simulation still has a sim-to-real gap and cannot alone certify real-world safety.

Continual learning provides cross-task knowledge retention but typically uses fixed update rules; AutoML searches training configurations but does not necessarily inherit the prior system; general recursive self-improvement frameworks often focus on digital environments. Physical RSI places autonomous inheritance, mechanism update, cross-task controlled gain, and physical grounding in the same judgment.

2.2 System Architecture and Improvement Loops

Execution Loop : Uses current model and harness to produce actions, writes trajectories to the experience library. Replanning or correcting the current action only improves the ongoing execution, not necessarily leaving an inheritable change.

Capability Update Loop : Reads experience, modifies skills, memory, or model, passes internal verification, and retains the change — effects appear in later tasks (e.g., turning a successful trajectory into a reusable skill, or fine-tuning a policy with new interaction data).

Meta-Update Loop : Modifies the improvement procedure itself — e.g., first localizing a failing module then modifying prompts, adjusting training methods when imitation learning plateaus, or changing candidate filtering mechanisms. Changes to harness adaptation and training phases can be checked by the internal verifier; changes to the verifier require external evaluation because it cannot alone adjudicate whether its own standards are more reliable.

The two paths influence each other: new skills alter the distribution of training trajectories; improved model capabilities change which harness modifications are worth trying. Safety must constrain all three loops — execution, capability update, and meta-update — not just the final version.

2.3 Six-Level Framework and Evidence Standards

Levels are determined jointly by the update object and the scope of gain, not by a single ladder of "automation degree."

L0 : No autonomous, retainable improvement loop. Only in-execution adjustments, or improvements still requiring human intervention, labeling, or version selection.

L1 : Same-task capability update. Fixed improvement rule; autonomously updates execution system and shows gain on the same task.

L2 : Same-task meta-update. Autonomously modifies and inherits the improvement procedure within the same task scope.

L3 : Cross-task capability update. Retained capability changes support different tasks; improvement rule may remain fixed.

L4 : Cross-task meta-update. Retained changes to the improvement procedure itself yield cross-task gains.

L5 : Independently verified recursive gain. Builds on L4; successively compares new vs. old improvement procedures, proving the new procedure produces better subsequent improvements under matched starting version, experience, and budget.

Task identity is defined by the required skill sequence. Changes in cup position or scene difficulty with unchanged skill sequence may still be the same task; adding a new skill or recombining/reordering existing skills constitutes a different task. Simulation and real-robot records are evidence sources but do not automatically change task identity.

Component ownership depends on usage, not name: a world model used at execution time belongs to the execution system; a world model used only for training environments belongs to the improvement procedure. Repeatedly training a reward classifier or value network by a fixed rule does not automatically constitute a meta-update; the procedure's own objectives, update rules, curriculum, or selection mechanisms must be changed and retained.

For example, Eureka optimizes rewards for 29 tasks separately — quantity does not equal retaining a single improvement mechanism and achieving cross-task recursive gain. ASPIRE uses a skill library acquired on short-horizon tasks for unseen long-horizon compositions, providing cross-task evidence, but this still differs from proving L5. Levels classify specific workflows and evidence, not permanent labels for method names.

Harness Adaptation of Physical RSI

3.1 Feedback Interpretation and Update Target Identification

Harness adaptation begins by answering "where did the failure occur?" Visual keyframes, force/contact signals, invocation logs, workflow node records, and success-failure contrasts help form causal hypotheses; prior modification history hints at recurring issues.

Diagnosis must map to editable components: instruction error vs. failure to follow correct instruction; reasoning deficit vs. missing perception tool; need to modify skill code, interfaces, parameter exposure, memory, or invocation order. The modification scope and applicability conditions should then be delimited, distinguishing local fixes from broadly reusable updates.

The paper surveys robot works such as ASPIRE and HarnessPAI, and digital-agent methods like ExpeL, AutoGuide, and HarnessFix. Digital methods offer transferable diagnosis and modification mechanisms but should not be misrepresented as proven evidence of real-robot recursive improvement.

3.2 Memory, Skill, and Workflow Adaptation

Memory Update : Beyond adding reflection text, includes organizing applicability contexts, updating outdated rules, and verifying physical explanations. PhysMem builds hypotheses from interaction and verifies applicability; KnowBody updates body-relevant relations and checks dependent knowledge.

Skill Update : Can repair existing programs, adjust interfaces, or distill new skills from successful solutions. ASPIRE repairs robot programs based on primitive operation feedback; ViReSkill saves successful plans and control code for reuse. New resources must actually enter subsequent execution, not just be recorded.

Coordination Update : Handles dependencies between shared functions and their callers. Introducing a common skill requires synchronizing its users; deleting redundant or stale abstractions must check dependencies. Otherwise local fixes cause regressions elsewhere.

Workflow Update : Changes when skills are invoked, how they hand off, how subgoals are advanced, and how recovery and stopping work. Differs from temporary replanning for the current task: adopted rules must be used in later executions. BATON emphasizes invocation and handoff conditions; RoboFoundry continuously adjusts phase handling, subgoal construction, and replanning cadence.

3.3 Adoption Mechanisms and Improvement History

Candidate updates are checked for intended effect and impact on existing capabilities. Re-running failure cases verifies fixes; testing shared functions and their callers detects regressions. Only changes accepted and entering subsequent execution constitute inheritance.

Adoption is not permanently valid. Later interactions may weaken confidence in a physical explanation; body estimate changes may invalidate dependent rules, requiring re-verification or suspension.

Improvement history should retain both successful and failed attempts, including raw observations, modified objects, diffs, rationales, verification results, and adoption decisions. Rejected updates inform future diagnosis. Recording "which method produced what modification" also grounds further improvement of the modification method itself.

3.4 Meta-Improvement of Harness Adaptation Mechanisms

Capability updates modify skill library or memory content; meta-improvements modify the methods that produce those resources — e.g., adjusting diagnosis strategy, memory construction operations, candidate generation prompts, or optimizer-side programs — so that later harness fixes become more effective.

Digital systems like STOP and Hyperagents demonstrate editable improvement procedures; Promptbreeder and MemSkill focus on prompt generation and memory construction. They offer methodological inspiration for physical settings, but evidence sources must be distinguished.

A new method cannot be judged better merely because it "generated different modifications." One must compare old and new methods from the same starting point, experience, and resources, examine gain vs. cost, and confirm the new method actually drives subsequent improvements after adoption. If both runtime components and adaptation methods change simultaneously, attribution must be disentangled; procedure evolution does not guarantee effectiveness under every setting.

Model Training of Physical RSI

4.1 Data Preparation

Data preparation links experience generation and learning selection. Real-robot practice can target weak capabilities; simulation generators extend tasks, scenes, rewards, and difficulty; real-scene reconstruction provides spatial context for sim-to-deploy. Selection mechanisms decide which successful trajectories serve imitation, which failures serve value learning or recovery, and which states to collect next.

Shared control, active help-seeking, and corrective demonstrations improve learning efficiency, but if retained updates depend on critical human corrections, they cannot be claimed as autonomous improvement under this framework. Increased data volume does not automatically indicate mechanism progress; only when the sampler, curriculum generator, or selection rule is modified and used for later learning does it potentially support a meta-update judgment.

4.2 Training Methods

Imitation Learning : Learns actions from demonstrations, teacher targets, or verified autonomous trajectories. Behavior cloning, diffusion and flow-matching action generation, policy distillation — all can serve as capability update paths. Modifying the learning objective may touch the mechanism layer, but evidence of physically grounded, sustainably inherited mechanism updates remains limited.

Self-Supervised Learning : Constructs training signals from observation and trajectory structure — reconstruction, latent action, dynamics prediction. Updating a world model can generate imagined trajectories for later policy training; fast weight adaptation at execution time changes the current context, but whether it becomes a retainable update requires separate judgment. Digital methods that search self-supervised objectives do not by themselves form a physical robot improvement loop.

Reinforcement Learning : Uses interaction outcomes to guide policy or value updates; can also train lightweight controllers outside a frozen policy. Reward code, training distribution, and update rules can all become mechanism modification targets. The key distinction: learning parameters by a fixed algorithm vs. autonomously changing and retaining the algorithm or rules used for subsequent learning.

4.3 Learning Models

Vision-Language-Action (VLA) Models : Map multimodal observations and instructions to robot actions. Updates can target action head, adapters, backbone, or the whole model; alternatively, keep the foundation model frozen and train only an external controller. Capability growth must be attributed to the component that actually changed.

World Models : Predict future states conditioned on actions, used for planning, action selection, or imagined training. A frozen model supporting repeated optimization vs. continuously updating the predictor with new physical trajectories are two distinct paths. Visual realism of predictions does not guarantee accuracy of contact, dynamics, or action outcomes; usage at execution vs. training time also determines framework ownership.

Reward Models : Evaluate progress, success, or behavior quality. Can be fixed to provide signals for policy learning, or updated with feedback and used to re-score historical data. Changed reward estimates affect later training, but refitting the estimator by a fixed rule still does not directly prove the improvement procedure underwent a meta-update.

4.4 Knowledge Transfer and Training Procedure Improvement

Cross-task transfer can reuse policy initialization, skill modules, action representations, expert routing, or controller compositions; replay, modular adaptation, and parameter merging help retain old capabilities. One must show that source-task retained changes are actually used by the target task and contribute gain, not merely report merged results from independently retrained tasks.

Data preparation mechanisms can adjust demonstration selection, data mixing ratios, simulation distributions, and collection focus based on learning feedback. Training mechanisms can modify reward code, update rules, training recipes, or experiment organization. Meta-updates must not only occur but be retained and used for subsequent training.

Physically grounded, inherited mechanism modifications can support L2 or L4 judgments in same-task or cross-task scopes; L5 further requires controlling starting capability, experience, and budget to prove the new mechanism produces more effective subsequent improvements than the old one. More data, stronger initial models, or extra compute cannot substitute for this comparison.

Safety and Control of Physical RSI

5.1 Safe Exploration and Continuous Adaptation

A safe policy after training does not guarantee safety during experience collection. Safe RL constrains expected cost, but average cost control may still permit individual dangerous events; constraint meaningfulness depends on whether cost truly reflects physical harm.

Maintaining task performance does not equal maintaining safety. Old and current scenes must be checked separately; historical safety labels must be re-evaluated under current dynamics and sensing conditions. Parameter retention or anti-forgetting mechanisms do not automatically preserve all safety properties.

Safe exploration can restrict to regions with existing evidence, checking reachability, return paths, and control stability. However, system updates or environment drift may invalidate assumptions; safe regions may need to shrink. If no provably supportive trial exists, physical exploration should pause.

5.2 Safe Transfer and Update Acceptance

Capability transfer does not imply safety evidence transfers. A new encoder may break state mapping; a new interface may invalidate fallback controller actions, even if task-required skills are unchanged — re-checking is necessary.

Three distinct decisions must be separated: Internal Acceptance — whether a candidate is inherited; Independent Evaluation — whether it truly improves capability; Deployment Authorization — under what conditions and safeguards physical operation is allowed. Scoring above an unsafe baseline does not prove safety.

Deployment evaluation must cover the full combination of policy, observation processing, action interface, monitors, and fallback controllers, checking sensor error, actuator limits, control timing, and handover states. Simulation feedback satisfies the physical grounding definition, but does not alone approve real-robot deployment.

5.3 Operational Assurance Under Environment Drift

Control barrier functions, reachability methods, predictive action filtering, and safety shields can limit actions or switch to fallback controllers, but their guarantees depend on valid models, uncertainty bounds, and control feasibility. Removing the filter at deployment loses its direct execution constraint.

Failure prediction only warns; it does not automatically guarantee safe intervention. Calibrating false-alarm rate does not guarantee miss rate or recovery success. Policy and encoder updates shift monitor input distributions, requiring recalibration; simply raising alarm thresholds to reduce interventions may mask risk.

Logs must distinguish proposed actions from executed actions, preventing frequent safety corrections from wrapping an unsafe policy as success. Recovery needs reachable safe states and effective controllers: rolling back to an old policy cannot reverse mechanical wear or friction changes. Detection error, response latency, and recovery feasibility must be jointly checked.

5.4 Supervision of Evolving Improvement Procedures

The improvement procedure determines what is tried next, how interventions are interpreted, and which updates are accepted; its safety evaluation must therefore examine the trial process that produced the policy and the successor systems, not just the final policy.

Internal verifiers must not relax protected safety requirements, nor evaluate themselves solely by their own revised standards. Modifying the verifier requires external evaluation comparing systems retained by different verifier versions. Existing formal results cover only specific structured updates and do not generalize to arbitrary self-modifying code.

When the procedure reads external logs or modifies code, external content must not gain authority to disable stop mechanisms or rewrite safety requirements. Human emergency stops, operation authorizations, training corrections, and version selections must be separately logged: retaining human safety control does not equate to counting human-driven learning as autonomous inheritance.

5.5 Cross-Generation Safety Evidence

Each system generation should link version, physical conditions, intervention records, and outcomes to verify whether old evidence assumptions still hold. A single-round risk bound is not a lifetime guarantee; while each round's bound holds for the adaptive process, multi-round violation probability is bounded by the sum of per-round bounds — risk accumulates.

Rejected trials, recoveries, and resets also incur physical exposure; hardware wear alters subsequent conditions. Risk during development of a new improvement procedure and risk during its later use should be accounted separately.

Erroneous update acceptance and actual physical damage are distinct events. Repeatedly using test feedback to select candidates creates adaptive selection bias; even if test tasks and scoring code are not directly read, final evidence independence is not guaranteed — new tests or statistical corrections are needed.

Safety evaluation should report violation frequency and severity, near-misses, misses, response latency, and recovery success rate, covering both immediate post-update performance and re-testing under old conditions. Safe operation and recursive gain require two compatible but non-substitutable evidence streams.

Verification and Evaluation of Physical RSI

6.1 Candidate Update Verification

Internal verification has three basic layers: implementation validity (syntax, interfaces, output formats); fix validity (re-execution on matched failure cases); capability retention (re-testing existing successful tasks and dependent knowledge). Runnable does not equal problem solved; solving the original case does not guarantee no harm to other tasks.

When the verifier itself changes, accepting more candidates does not mean verification quality improved. One must evaluate systems retained by old and new verifiers using the same external tasks and scoring criteria. Candidate-stage development evidence cannot replace protected independent outcome evaluation.

6.2 Improvement Outcome Evaluation

Performance and Retention : Compare before/after and across versions; check whether gains accumulate, persist, or temporarily dip; test with initial conditions not used in debugging or candidate selection.

Attribution and Transfer : Match initial data, model, training recipe, and trajectory budget; use component freezing, empty-skill-library baselines, or version ablations to separate harness, model, and procedure contributions; transfer tests should confirm task change via skill sequence.

Efficiency and Autonomy : Beyond task success, report robot time, GPU time, token consumption, data preparation, training, reset, and goal-reaching costs; explicitly disclose human setup, corrections, takeovers, and version selections. Closing human help during evaluation does not prove the improvement phase was human-free.

Key Counterfactual for Recursive Gain : Do not directly compare final scores of generation N vs. N+1. Instead, let old and new improvement procedures start from the same generation, same experience, matched tasks and budget, produce subsequent systems, then independently evaluate cross-task effects. Safety rules, stop conditions, reset arrangements, and operation authorizations must also match. If the evaluation task set expands, re-evaluate early versions or use a fixed anchor subset for comparison.

6.3 Benchmarks and Evaluation Resources

Resources organized by evaluation focus: - Task learning and generalization: Meta-World, RLBench, CALVIN, LIBERO. - Robustness and real environments: SIMPLER, LIBERO-PRO, RoboArena. - Success judgment and feedback reliability: FailBench, RoboRewardBench. - Robot programming/training development: CaP-Bench, RoboCoach-Bench. - Digital recursive improvement: AI4AI-Bench, RSIBench-Data, RSI-Exam (evaluate training algorithms, data strategies, executable method modifications).

Each covers relevant capabilities, yet none yet forms a unified benchmark for physical recursive improvement that jointly tests autonomous inheritance, cross-task mechanism improvement, physical feedback, and safety cost.

Challenges and Future Directions

Learning from Uncertain Physical Experience : Partial observability, sensor noise, and stochastic outcomes make success judgment and failure attribution unreliable. Erroneous feedback that modifies the training procedure may amplify in later rounds. Future work must link feedback credibility to inheritance decisions, not just collect more trajectories.

Causal Attribution Across Harness Components : Memory, skills, and workflows interdepend; surface failures may stem from upstream defects; simultaneous multi-component changes hinder contribution separation. Need controlled experiments within limited trial budgets to identify causal dependencies and justify modification targets and scopes.

Non-Stationarity of the Improvement Loop : Update effectiveness depends on the system configuration, capabilities, and physical conditions at that time. Later component changes or environment drift invalidate old conclusions. Must track evidence dependencies, identify what must be re-verified vs. reused, and limit re-verification cost.

Reward Hacking and Evaluation Leakage : Inaccurate success detectors and reward models may drive optimization of scores rather than true physical outcomes. Protecting test tasks from being read is insufficient; repeated feedback-based candidate filtering also breaks independence. Directions include independent physical outcome verification, controlled feedback disclosure, and final tests with new evidence or statistical correction.

Safety in Recursive Updates : Safety guarantees for arbitrary improvement procedure updates remain open; full re-verification every version may exceed budget and add risk. Authors propose selective re-review: identify failed assumptions, reuse still-valid evidence, concentrate trials on unresolved safety issues, and retain protected requirements and effective stop capability.

Building Recursive Gain Benchmarks : A stronger successor system may simply have more experience or a better starting point. Benchmarks should trace the procedure's autonomous modifications, adoptions, and actual use; compare consecutive procedure versions from identical starting points and experience; match tasks, resources, and safety conditions; avoid mistaking data accumulation for mechanism progress.

Conclusion

The paper advances robot autonomy from "can execute" to "can improve," then asks "can it improve how it improves?" The L0-L5 framework organizes existing research by update object, task scope, and independent evidence, connecting harness adaptation and model training paths with safety, inheritance, and evaluation.

It does not declare that robots have achieved continuous recursive evolution. On the contrary, the authors highlight that existing evidence falls short of strict physical recursive self-improvement. The key is not showing version scores rising, but proving that mechanism modifications themselves, under controlled starting points and budgets, make subsequent cross-task improvements more effective.

Key takeaways: Capability improvement, mechanism improvement, and verifiable recursive gain are three distinct levels. More reliable physical feedback, clearer causal attribution, safety evidence that remains valid as system and environment change, and dedicated benchmarks are necessary conditions for reaching higher levels.

Paper link: https://arxiv.org/abs/2610.18842

Full PDF: https://arxiv.org/pdf/2610.18842

Code and resources: https://github.com/SAIL-Research-Lab/Physical-RSI

Code example

来源：专知
本文
约10000字
，建议阅读
15
分钟
机器人表现越来越好，并不自动意味着它已经越来越擅长改进自己。