A Comprehensive Survey of Self‑Evolving Agent Systems by the Father of Generative AI

Core definition

At time t an agent is defined as A_t = (θ_t, Σ_t) where Σ_t = (p_t, m_t, T_t, g_t). θ denotes the base‑model parameters (the cognitive core). Σ is the operational scaffold (the “harness”) consisting of:

p : prompt or system instruction

m : memory mechanism and its retrieval/update strategy

T : tool set and invocation interface

g : control logic (routing, scheduling, safety constraints)

Classification framework

The survey splits self‑improvement into two evolution paths—base‑model improvement (θ) and scaffold improvement (Σ)—and then refines each path by the source of the learning signal.

Path 1 : Base‑model improvement (θ)

θ‑improvement is organized into three signal‑based loops.

5.1 Intrinsic data generation (S≈D)

Models extract training data from their own weights, producing instruction‑answer pairs, reasoning traces, or execution logs. Representative works include Self‑Instruct , Evol‑Instruct , LMSI , Ladder , and TT‑SI (detects uncertainty at test time, generates data for blind spots, and performs instant LoRA fine‑tuning).

Risk : model collapse and knowledge bubbles—retraining on self‑generated data can amplify bias and narrow the solution space. Mitigations mentioned are retaining real data as a foundation, using external validators or theorem provers, diversifying data pools, and conducting human audits.

5.2 Intrinsic evaluation feedback (S≈e)

Models act as their own judges, producing judgments instead of examples. Three families are identified:

Rubric feedback : scoring by explicit criteria (Constitutional AI, Meta‑Rewarding, Self‑Evolved Reward Learning).

Consistency feedback : sampling multiple outputs to derive consensus, entropy, or self‑certainty (TTRL, INTUITOR).

Corrective feedback : generating critiques and revised drafts (SELF, RISE, Reflect‑Retry‑Reward).

Critical weakness : same‑source coupling between evaluator and policy can reward superficial compliance rather than genuine improvement; consistency signals may amplify systematic errors.

5.3 Extrinsic exploratory experience (S≈τ)

Signals are grounded in action outcomes and split into two environment types:

Real‑task environments : programmatic validators (unit tests such as Agent‑RLVR), learned reward models (WebRL, UI‑Genie), self‑generated tasks (Absolute Zero).

Simulated agent environments (World Models) : learning dynamics then “mental rehearsal” (WebEvolver, WMPO).

FM‑era specific pitfalls : language‑level reward hacking, capability degradation from narrow‑reward RL, and hallucination dynamics where generative simulators produce plausible but erroneous transitions that policies exploit.

Path 2 : Scaffold improvement (Σ)

This path keeps the base‑model weights unchanged and evolves the scaffold through four layers of increasing intervention depth.

6.1 Prompt – four generations of increasingly “nutrient‑rich” signals

Scalar feedback optimization : scores only, using search (APE, OPRO, RLPrompt, InstructZero).

Qualitative feedback refinement : text critiques drive directed edits (Self‑Refine, Reflexion, ACE).

Population evolution : prompts act as genes undergoing selection, crossover, mutation (EvoPrompt, Promptbreeder, GEPA).

Text gradients : formalizing critiques as directional update vectors (APO, TextGrad, MetaTextGrad, SkillOpt).

The signal flow progresses from scalar → critique → population fitness → structured direction, reducing reliance on heuristics and increasing automation.

6.2 Memory – from static cache to autonomous engine

Three dimensions are distinguished:

Memory objects : explicit (processed trajectories, curated text, external knowledge – explainable, auditable) vs implicit (latent embeddings, KV‑enhanced – compact but hard to debug).

Memory structures : flat (cheap, recent‑bias) → hierarchical (compress long history, fragile) → graph (supports multi‑hop reasoning, complex) → vector retrieval (semantic recall, but similarity ≠ usefulness).

Memory processing : signal‑driven CRUD – Create (selective distillation), Read (mixed heuristics + gated retrieval), Update (periodic rehearsal, iterative distillation), Delete (multi‑level pruning, hierarchical eviction), plus Select (who writes/reads) and Maintain (freshness vs forgetting).

Representative systems: Mem0, A‑MEM, Zep, G‑Memory, MemoryOS.

6.3 Tool – from “knowing how to use tools” to “tool‑governance meta‑cognition”

Dynamic routing : retrieval & graph routing (ToolNet, OrchDAG), policy‑learned routing (Tool‑Star, AGENTFLOW, ToolGen), proactive interactive routing (MCP‑Zero, Tool‑Planner).

Iterative refinement : generate‑execute‑revise loop (Voyager foundation, STELLA with dedicated critic, SkillWeaver distilling reusable web tools, DRAFT fixing tool documentation).

Autonomous creation : TOOLMAKER automates the pipeline from paper logic to dependency installation, interface generation, and testing; Alita/Code2MCP standardize codebases as services to prevent tool explosion.

6.4 Full scaffolding – treating the entire codebase as a mutable substrate

The deepest intervention formalizes scaffold update as Σ_{t+1} = I_{Σ_t}(Σ_t; S_t), where the improver I is itself part of Σ_t, leading to co‑evolution of improver and system. In practice this becomes “generate patch → validator gate (unit tests, regression, safety checks) → submit if passed”.

Representative systems: Darwin Gödel Machine, Huxley‑Gödel Machine, GödelAgent, Live‑SWE‑agent, AlphaEvolve/ShinkaEvolve, STOP (Self‑Taught Optimizer), Agent Symbolic Learning.

References and resources

Self‑Improvements in Modern Agentic Systems: A Survey
https://arxiv.org/abs/2607.13104
https://selfimproving-agent.github.io/
https://github.com/selfimproving-agent/awesome-Self-Improving-Agents