Trustworthy Recursive Self-Improvement in Large Models: Foundations, Risks & Evaluation

From Self-Adjustment to Recursive Self-Improvement (RSI)

AI systems can already use task experience and feedback to adjust themselves — for example, coding agents write experience into memory, modify prompts, or change tool-calling rules so that changes persist across tasks. However, such self-adjustment is not automatically RSI. The key distinction is whether the system can further improve the mechanism that produces these changes.

The research team proposes an operational definition and process framework for RSI, dividing the improvement process into four stages:

S1 — Diagnose system limitations

S2 — Generate candidate solutions

S3 — Evaluate candidate solutions

S4 — Implement updates that pass evaluation

On this basis, the system can recursively apply the same process to the update function itself, a process called Recursive Improvement (RI) . This creates two layers of change: the first layer modifies how the AI solves tasks; the second layer modifies how the AI produces those modifications. When the improvement mechanism itself becomes an update target, it affects not only current task results but also what future improvements will look like and by what standards they will be judged.

The team emphasizes that forming a recursive update loop does not guarantee the system will continuously grow stronger, nor does it prove trustworthy self-evolution.

Four Trustworthiness Themes

If a system can continuously modify itself, evaluation cannot focus only on task completion rates or single-test scores. The team distills trustworthy RSI requirements into four themes:

Genuine and sustained capability improvement. Higher scores do not necessarily mean real capability gains; a one-time gain may not persist. Evaluation must check for regression of existing capabilities and whether improvements endure.

Behavioral constraints, alignment, and fairness. Even as task capability grows, the system must continue to satisfy safety requirements and avoid unreasonable disparate impact. If an update changes original constraints, those requirements must be re-verified.

System security, resilience, privacy, and data governance. Researchers must check whether malicious content, poisoned memories, or sensitive information are inherited by later versions, and whether the system can recover to a compliant state after attacks or failures. Risks can accumulate across versions.

Transparency and controllability. What changed, why an update was accepted, and whether problems can be traced and rolled back must be evaluable. If the improvement mechanism can modify the oversight process itself, humans must confirm that intervention capability remains effective.

These four themes merge into three analytical perspectives: improvement effectiveness , constraint preservation (covering behavioral constraints and system security), and effective oversight (covering transparency and controllability). They are not separate stages or independent checklists; together they ask whether capability gains, safety constraints, and human oversight can all be evidenced on a single continuous system trajectory.

Three Levels of Evaluation

The team further discusses three levels at which these requirements must be verified:

Single update: Does a specific modification deliver the expected benefit while satisfying necessary constraints?

Subsequent inheritance systems: Can those benefits and constraints persist in later versions? A single update passing tests does not guarantee long-term effectiveness; a change may improve short-term performance but introduce new issues or cause capability decay later.

The improvement mechanism itself: Does the modified updater actually produce better subsequent updates? Observing only the final system's performance makes it hard to attribute gains to the updater versus changes in task conditions, compute budget, or other factors.

For the improvement mechanism, the team suggests paired continuous evolution experiments from comparable initial conditions, controlling task sequences, permissions, and compute budgets, then independently evaluating later versions. Evaluation must track changes across multiple versions, observing whether capability gains are retained, constraints remain satisfied, and oversight stays effective.

Seven Open Research Challenges

The survey identifies seven directions for future work. Three are highlighted:

How to verify a self-modifying improvement mechanism? When the system can adjust its own evaluator or acceptance criteria, score improvements may stem from changed test rules rather than real capability gains. Prior verification conclusions may no longer apply; the reliability of the changed standards themselves must be re-examined.

How to understand long-term risk evolution? Issues introduced in one update may persist or interact with later changes. Single-update checks cannot fully capture risk evolution over long trajectories. Researchers need to track risk over extended system lineages and test whether existing mitigations remain effective.

Can trustworthiness assurance mechanisms themselves be improved? Evaluators, monitors, and safety guards may become modification targets. Improving them could help the system better identify risks, but weakening them could make later updates harder to evaluate correctly. Proving that changes to assurance mechanisms enhance trustworthiness rather than reduce safety remains open.

Additional factors influencing required verification intensity include the scope of what the system can modify, who controls evaluation feedback, and the consequences of accepted updates. For example, reversible software changes and irreversible real-world automated actions demand different risk assessment approaches. Larger modification scope and harder-to-reverse external effects require stronger independent verification, explicit authorization, and effective intervention.

Conclusion

RSI research is shifting from "how to make systems produce better results" to "how to prove the entire improvement process is trustworthy." Models, memory, tools, and workflows can all become update targets; when the mechanism that produces updates is also modifiable, existing evaluation standards and safety guarantees face new challenges.

This survey does not claim trustworthy RSI has been achieved. Instead, it maps current progress while clarifying risks and evaluation gaps. Its emphasis is not merely on continuous improvement, but on building an evidence system that can verify whether improvements are real, durable, and compliant.

To prove RSI has trustworthy long-term improvement capability, researchers must jointly verify three things: whether capability truly improves, whether behavioral and safety constraints continue to be satisfied, and whether human oversight and intervention remain effective. Rather than rushing to discuss when AI might achieve unlimited self-evolution, a more practical question comes first: when AI starts modifying the next generation of AI systems, can we prove it makes them not only stronger but also reliably controllable throughout continuous improvement?