Skip to content
SkillMaster: LLM Agents That Autonomously Evolve Their Own Skills via Counterfactual Rewards

SkillMaster: LLM Agents That Autonomously Evolve Their Own Skills via Counterfactual Rewards

Tsinghua University's SkillMaster framework teaches LLM agents to create, update, and retain reusable skills through reinforcement learning, using counterfactual utility rewards and DualAdv-GRPO to jointly optimize task execution and skill management, achieving 98.7% on ALFWorld and 82.0% on WebShop with Qwen2.5-7B.

Machine Heart
Machine Heart
Machine Heart

As recursive self-improvement (RSI) gains attention, skills that distill interaction experience into reusable procedures become a concrete path for continual agent evolution. However, existing methods rely on external teacher models, handcrafted rules, or auxiliary modules for skill creation and updates; the executing agent learns to use skills but is rarely trained to improve them.

SkillMaster, accepted at NeurIPS 2026 (arXiv:2605.08693, GitHub: sduyangmin/Skill-Master), addresses this by making "how to evolve skills" a learnable capability of the agent itself. The core insight: a valuable skill edit should make previously failed similar tasks easier to succeed and previously successful tasks completable in fewer steps. These two observable effects are turned into learning signals for skill editing, so skill evolution and task execution enter a single reinforcement learning loop.

Method Framework: Joint Learning of Task Execution and Skill Evolution

SkillMaster solves three questions — what edits to propose, how to evaluate their value, and how to jointly optimize both decisions — via three components:

Trajectory-based skill introspection: The model receives the task goal, retrieved skills, action-observation trajectory, and final feedback. It then calls propose_skill, update_skill, or keep_skill tools to create, revise, or retain skills. For example, discovering a missing search strategy triggers a new skill; observing a conflict between feedback and an existing rule triggers a revision.

Counterfactual skill utility reward: For each candidate edit, a set of related probe tasks (sampled from the training pool by category, product type, or semantic similarity) is run with both the original and the temporarily edited skill library. The downstream change — failures turning into successes, or successful tasks requiring fewer steps — is aggregated into a utility score. Only edits with positive utility are written to the persistent skill library.

DualAdv-GRPO: Task execution relies on environment outcome rewards, while skill management uses tool-call validity and downstream utility rewards. Because the two reward types differ in scale and meaning, DualAdv-GRPO computes separate advantages for each and assigns them to the corresponding generated tokens, updating a single policy that learns both effective task execution and skill improvement.

Experimental Results

Experiments use Qwen2.5-7B-Instruct on three benchmarks:

ALFWorld (six household task types): SkillMaster reaches 98.7% success rate, +8.8 pp over SkillRL. Four task types hit 100%, the other two exceed 95%.

WebShop (multi-step product search and purchase): 82.0% success rate, +9.3 pp over SkillRL.

SearchQA (seven single- and multi-hop QA datasets): Overall 48.0%, +0.9 pp over SkillRL, leading on five of seven datasets.

Ablation (Figure 3 left) shows removing the utility reward, merging advantage normalization, or random probe selection all degrade performance, confirming the contribution of utility feedback, separate advantage estimation, and relevant probe selection. During ALFWorld training, create/update operations dominate early, while keep-skill proportion rises from ~81.25% to 97.66% later, reflecting a shift from skill acquisition to retention. A fixed-policy test on WebShop starting from an empty skill library shows success rate climbing from 65.63% to 71.88% when skill construction and retrieval are both enabled, demonstrating that the trained policy can generate and reuse useful skills in new interactions.

Case Analysis: Creating and Revising Skills from Execution Evidence

In an ALFWorld cooling task (Figure 4), the agent wasted steps searching low-probability cabinets and drawers, ultimately failing. During introspection it identified a missing search-priority rule and created a new skill directing future searches to high-probability locations relevant to the target object.

Another case shows skill revision: the ALFWorld interface requires heating while holding the target object, but an old skill instructed placing the object in the microwave first. The model revised the precondition based on trajectory feedback. Counterfactual probe tasks show the heating step dropping from 8 steps to 2, eliminating six redundant operations — a direct measure of downstream utility.

Outlook: Toward Recursive Self-Improvement via Skill Evolution

SkillMaster explores one RSI path by treating skills as reusable knowledge carriers. The agent detects knowledge gaps or inefficient strategies during interaction, edits skills to change future behavior, and learns better editing decisions from actual outcomes. Task execution and skill improvement are interleaved, letting the agent both accumulate skills and improve its ability to improve skills. Validated in structured interactive environments and search-augmented QA, future work targets open environments with more efficient probe selection, lower-cost utility evaluation, and extensions to skill deletion, cross-agent sharing, and continual learning under distribution shift.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Reinforcement LearningLLM agentsRecursive Self-ImprovementNeurIPS 2026autonomous skill learningcounterfactual rewardDualAdv-GRPOSkillMaster
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.