Why Skills Improve Over Iterations: SkillHone Lets Agents Remember Each Optimization

Problem with Existing Skill Evolution Methods

Current approaches either generate new skills from task descriptions (Skill‑Creator) or iteratively modify existing skill candidates (Hermes‑SE). They discard the optimization process, preserving only the final version and losing the rationale behind each change.

Modeling the Optimization Process

SkillHone treats each optimization step as a first‑class artifact. Every round records diagnosis, candidate edits, sanitized evaluation evidence, and the final decision in a persistent decision history that spans the entire skill repository—including SKILL.md, scripts, reference materials, and templates. This history enables subsequent agents to understand why a change was made and to reuse past experience.

Persisting the Optimization Process

Two interlinked repositories are defined:

Skill repo : contains the skill description, scripts, data, and templates.

Skill‑Eval repo : holds probes, ground‑truth answers, validators, execution traces, and regression tests.

At the start of each evolution round, the evaluation side runs the current skill on a regression suite and produces a sanitized report. The optimization side edits the skill based on this report and existing history; the evaluation side then re‑tests the candidate on the baseline version. The system decides to accept, continue modifying, or reject the candidate and writes the evidence back to the history.

Roles are isolated: the optimization agent can modify the skill but cannot see unsanitized answers, while the evaluation agent can run tests and view traces but cannot write to the skill repository. A scheduler dynamically creates sub‑agents for diagnosis, development, review, execution, and reporting, requiring only the ability to spawn child agents. This design is compatible with runtimes such as Claude Code, Codex, and Hermes.

Repo‑Level Updates

Unlike prior methods that edit only SKILL.md, SkillHone expands the optimization target to the full repository, allowing modifications to scripts, templates, and auxiliary files. This broader scope enables more substantial improvements and reduces the need for repeated manual edits.

Precise Rollbacks

When a candidate’s overall score degrades, traditional systems discard the entire version, losing any beneficial changes. SkillHone’s decision history pinpoints the specific modification that caused regression, allowing precise rollback of only the problematic part while preserving effective improvements. In a five‑round optimization trace, adding DuckDuckGo fallback and error handling raised probe accuracy from 30 % to 60 %; subsequent adjustments raised it to 70 %.

Experiments: Performance, Transfer, and Ablation

SkillHone was evaluated on the GAIA and WebWalkerQA‑EN benchmarks using the Qwen3.6‑35B‑A3B execution model. Results:

GAIA: 64.6 %

WebWalkerQA‑EN: 66.4 %

These scores outperform commercial deep‑research agents by 15.8 and 3.2 points respectively, and surpass Hermes‑SE by 14.2 and 13.4 points. After migrating the same skill package to Claude Sonnet 4.6 without re‑optimizing, GAIA performance rose to 72.4 %, demonstrating transferability across execution models.

Ablation studies show that removing the decision‑history component drops GAIA and WebWalkerQA‑EN scores by 13.4 and 10.9 points, while eliminating role isolation reduces them by 6.4 and 5.3 points, confirming the importance of both components.

References

Paper: SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History (arXiv:2606.08671) – https://arxiv.org/abs/2606.08671 Code repository:

https://github.com/Tencent/SkillHone