How Can an Agent Strengthen Itself? A Full Overview of Skill‑to‑Weight Evolution

Definition of Self‑Evolution

Traditional foundation models are trained once and deployed statically, lacking continuous learning during real interactions. Self‑Evolution (Agentic Self‑Evolution) enables AI systems to behave like living organisms: they continuously interact with the environment, collect feedback, accumulate experience, and reconstruct themselves, achieving dynamic adaptation and capability growth.

Core Mechanisms

Dynamic Adaptation Loop

The heart of self‑evolution is a closed‑loop mechanism: Execute → Feedback → Self‑Adjust → Execute. This loop lets the agent improve its behaviour during runtime based on observed outcomes.

Multi‑Dimensional Evolution Targets

Evolution is not limited to model‑weight updates. It also covers non‑parameter components such as Prompt , Skill , Memory and Workflow , providing higher flexibility and self‑driven capability.

Skill Evolution Methods

Trace2Skill

Trace2Skill extracts transferable skills from large batches of agent execution trajectories. Its workflow consists of:

Batch trajectory generation: An initial skill (human‑written or LLM draft) generates many task trajectories, which are classified as successful or failed.

Parallel multi‑agent patch proposal: A set of analysis sub‑agents processes each trajectory in parallel, producing skill‑modification patches without interference. In practice, a 122B LLM generated 200 trajectories containing more than 50 rounds in under two hours.

Conflict‑free patch integration: Hierarchical merging and inductive reasoning combine the massive patches into a coherent, conflict‑free skill set.

EvoSkill

EvoSkill implements a three‑sub‑agent system (Executor, Proposer, Builder) that iteratively refines skills:

Executor: Executes the current skill on a task, producing a full execution trace.

Proposer: Analyses the trace, identifies failures, and suggests concrete optimisation directions.

Builder: Implements the proposed changes as code or rule updates.

This design ensures clear directionality, avoids blind updates, and enables quantifiable, feedback‑driven optimisation.

SkillOpt

SkillOpt treats skill optimisation as a text‑parameter training problem. A specialised “optimiser” agent performs small‑batch reflective updates on successful and failed cases, applying learning‑rate‑like constraints and strict validation gating. The process mimics SGD‑style optimisation, yielding stable and interpretable skill evolution.

CoEvoSkill

CoEvoSkill introduces cooperative‑competitive co‑evolution. Multiple skill agents are dynamically activated based on task context; their proposals are aggregated via voting or LLM arbitration. A step‑level advantage signal derived from repeated environment states (Anchor States) provides dense supervision, solving the credit‑assignment problem in long‑horizon tasks.

SkillRL

SkillRL tightly couples reinforcement learning with skill evolution. During RL training, successful trajectories are distilled into reusable skills, while failed trajectories generate avoidance knowledge. A hierarchical Skill Bank stores generic, task‑specific, and error‑type skills, enabling recursive skill‑augmented RL loops that continuously improve both policy and skill library.

D2Skill

D2Skill employs a dual‑granularity skill model plus contrast‑driven utility evaluation. It distinguishes between high‑level task‑guiding skills and low‑level step‑correcting skills, and uses parallel sampling against a baseline to compute utility differences that shape rewards and policy updates.

Group‑in‑Group Policy Optimization (GiGPO)

Both SkillRL and D2Skill use GiGPO, an extension of GRPO that retains the critic‑free advantage while adding a step‑level advantage layer.

Episode‑level Advantage: Within each task group, total returns of full trajectories are normalised (group‑wise) to provide a global good‑vs‑bad signal.

Step‑level Advantage: Anchor States (re‑visitable environment states) are identified via offline hash aggregation. Within each Anchor State group, discounted rewards are normalised to rank actions locally, delivering dense step‑wise supervision and solving credit‑assignment.

Fusion: The two advantages are linearly combined (default 50 % each) and fed into the clipped PPO objective.

GiGPO incurs almost no extra memory or compute overhead because it reuses rollout data and avoids an additional critic network, making it well‑suited for long‑sequence LLM‑agent training where state revisits are frequent.

Meta‑Evolution (HyperAgent)

When the improvement process itself becomes the optimisation target, the system enters a Meta‑Evolution regime. A HyperAgent consists of a Task Agent (the worker) and a Meta Agent (the optimiser). The Meta Agent can rewrite the Task Agent’s code and also improve its own optimisation logic, leading to emergent capabilities such as persistent memory and performance tracking without explicit design.

Experiments show that HyperAgents trained on paper‑review and robot‑manipulation tasks transfer effectively to IMO math scoring, achieving performance comparable to specialised models (e.g., DGM‑H imp@50 ≈ 0.630).

Open Challenges

Direction control (over‑fitting risk): Single‑turn trajectories are noisy and may cause skill patches to over‑fit rare bad cases, degrading generalisation.

Quality instability (lack of verification): Without a strict quantitative evaluation loop, it is hard to distinguish genuine skill improvement from environmental noise, threatening service stability.

Addressing these issues requires robust data‑flow feedback, systematic validation, and controllable evolution mechanisms.

References

Self‑Improvements in Modern Agentic Systems: A Survey

Trace2Skill: Distill Trajectory‑Local Lessons into Transferable Agent Skills

EvoSkill: Automated Skill Discovery for Multi‑Agent Systems

SkillOpt: Executive Strategy for Self‑Evolving Agent Skills

CoEvoSkills: Self‑Evolving Agent Skills via Co‑Evolutionary Verification

SkillRL: Evolving Agents via Recursive Skill‑Augmented Reinforcement Learning

Dynamic Dual‑Granularity Skill Bank for Agentic RL

Group‑in‑Group Policy Optimization for LLM Agent Training

AgentEvolver: Towards Efficient Self‑Evolving Agent System

Self‑Distilled Reasoner: On‑Policy Self‑Distillation for Large Language Models