Why Agent Skills Need Self‑Evolution: A Survey of 19 Frameworks and 10 Benchmarks

You give an Agent a skill—e.g., "automatically handle customer tickets"—and it works well at first, but later encounters new ticket types and starts failing. Should you edit it manually or let it learn to fix itself, and how can you be sure the fix is safe?

The survey, a joint effort by Rutgers University and UNC Charlotte, is the first systematic overview of Agent skill evolution and evaluation. It covers 19 evolution methods and 10 evaluation benchmarks, and its most striking conclusion is that no benchmark currently tracks whether a skill truly improves after multiple rounds of feedback; the evaluation landscape is still stuck at a simple "pass/fail" stage.

What a skill is and why it can't just be written once

An Agent skill is not a prompt nor an API call; it is a structured package that defines when to trigger, how to execute, when to terminate, and whether it can be composed with other skills.

When the skill library contains only a few entries, manual maintenance is feasible. As the library scales, manual upkeep becomes unrealistic—an outdated or defective skill propagates errors downstream. Therefore, skills must continuously evolve rather than being written once and left unchanged.

Four evolution pathways, different signal sources

Execution feedback : error output or runtime exception from a single run; step‑level granularity; examples: SkillForge, CoEvoSkills, Skills‑Coach, Ctx2Skill, AutoSkill, SkillClaw, EmbodiSkill.

Trajectory distillation : success/failure trajectories over multiple runs; sequence‑level granularity; examples: SPARK, Trace2Skill, Memento‑Skills, XSkill.

Compression & augmentation : overall skill‑library structure (overlap, conflict, gaps); library‑level granularity; examples: SkillNet, SkillX, SkillReducer, SkillFoundry.

Reinforcement learning : task‑completion reward; task‑level granularity; examples: D2Skill, SkillRL, SkillOS, Skill1.

These four routes are not mutually exclusive but represent the most mainstream design choices.

Key finding: separating “failure diagnosis” from “skill rewrite” yields markedly better cross‑task performance. SkillForge and CoEvoSkills first use a validator for root‑cause analysis, then let the model rewrite the skill, outperforming AutoSkill and SkillClaw, which modify directly on raw trajectories because the raw data contains too much noise.

Another notable trend is library‑level compression: when the skill library expands to hundreds or thousands, overlap and conflict cause redundant exploration and poorer generalisation. SkillX merges similar skills, splits complex ones, and evaluates generalisation, prioritising tools that are “under‑tested” or “frequently fail”.

Fundamental RL paradox: cannot tell whether skill improved or model got stronger

Standard RL rewards a single task, yet a skill’s true value lies in cross‑task reuse.

D2Skill runs each task twice (with and without the skill) and uses the success‑rate gap as a more stable reward signal. Skill1 goes further, employing a single policy to search, select, solve, and evolve skills.

All these methods rely on task‑level reward, which mixes skill quality with model capability, making it impossible to determine whether observed performance gains stem from skill improvement or a stronger underlying model; no existing approach cleanly separates the two factors.

Three structural blind spots in current evaluation

The survey’s systematic analysis of ten skill‑evaluation benchmarks uncovers three systemic problems:

No longitudinal tracking: No benchmark follows a skill across multiple feedback rounds, so we only know whether a single round “passes” but not whether the skill is truly improving over time.

Metrics are binary “pass/fail” only: They ignore token cost, latency, and error types—critical factors in real deployments. SkillsBench shows that selected skills raise average pass rate by 16 pp, but the lift varies from +4.5 pp in software engineering to +51.9 pp in medical domains, illustrating that a single pass‑rate metric hides large differences.

Security is checked only once: SkillGuardBench annotates 581 skill packages with three attack patterns (hidden overwrite, disguised transmission, remote hijack), yet current security audits treat skill safety as a one‑time gate rather than continuous monitoring. Uncontrolled self‑evolution can silently strip existing safety constraints, a risk the current evaluation framework cannot address.

Implications for building Agent systems

Skills are not static code; they must be treated as living assets that require continuous iteration. However, today’s evaluation infrastructure lacks even the most basic software‑engineering artefacts such as version numbers and regression tests.

The core judgment of the survey is that the skill ecosystem should be viewed as continuously evolving infrastructure rather than a static tool. Ongoing evaluation and evolution are essential for reliable use, control, and real‑world deployment, but current evaluation systems fall far short of supporting this need.

Paper title: Agent Skill Evaluation and Evolution: Frameworks and Benchmarks
Paper link: https://arxiv.org/abs/2606.11435v1
GitHub: https://github.com/Cassie07/AgentSkill_Survey