Teaching Agents to Evolve: The Hierarchical Skill Meta‑Evolving Framework HiSME

Gap in Static Skill Evolution

Static skill evolution suffers from two pain points. First, it struggles to adapt to diverse scenarios: tool‑calling tasks require precise API parameters and state tracking, while open‑world embodied tasks need long‑horizon action composition and failure recovery, making a single fixed strategy insufficient. Second, maintenance costs are unbounded; systematic bias in the evolution algorithm can repeatedly generate low‑quality skills, incurring extra cost for repair, rollback, or filtering.

HiSME: A Hierarchical Skill Meta‑Evolving Framework

HiSME treats skill evolution as test‑time marginal performance optimization. It does not modify the underlying LLM parameters; instead, it augments the fixed executor with a skill library that captures experience. If the skill‑evolution algorithm itself is sub‑optimal, higher‑order marginal gains can still be harvested from test‑time feedback.

HiSME structures the system into three layers: trajectories, skills, and meta‑skills.

Trajectory Layer : Agent executions produce trajectories evaluated on success rate, correctness, and computational cost. Any trajectory that can be evaluated can yield experience about what to do or avoid in the future.

Skill Layer : HiSME extracts candidate skills from single trajectories (extractor) and abstracts shared structures from multiple similar trajectories or existing skills (refactorer). To prevent noise, a bundle tester, credit assigner, refiner, and filter assess skill contracts, assign posterior credit, and decide whether to revise, disable, or retain each skill. Skill evaluation focuses on two dimensions: effectiveness (does the skill improve downstream utility?) and reusability (is the skill frequently retrieved or invoked?).

Meta‑Skill Layer : HiSME records not only skill performance but also which algorithmic component produced or modified each skill (extractor, refactorer, refiner). When a skill accumulates effectiveness and reusability feedback, the corresponding evolution strategy is evaluated, and a dedicated meta‑refiner summarizes this into a meta‑skill. Meta‑skills act as rules‑of‑thumb prompts, e.g., “extractor should only extract workflows with verifiable state or API preconditions” or “refiner should narrow trigger conditions when a skill misfires.” These textual rules are written back into the prompts of the extractor, refactorer, and refiner, guiding subsequent evolution.

Experiments: Performance Gains and Transferable Evolution Strategies

The paper evaluates HiSME on BFCL‑v3 multi‑turn tool‑calling and MineDojo open‑world benchmarks. Baselines include a no‑skill model, a static skill evolution baseline (SkillX), and two test‑time learning baselines (AWM, Memento). HiSME consistently outperforms all baselines. On MineDojo, HiSME also reduces token overhead because high‑quality skills cut down invalid exploration, failed recoveries, and redundant retries.

Ablation studies reveal that the refactorer is a critical component: reusable structures often require cross‑trajectory evidence rather than isolated insights. The extractor, refiner, and credit filter together form a closed loop for skill generation and maintenance.

Statistical analysis of the evolution process shows that, as training tasks increase, both HiSME and its static variant improve their skill libraries, but HiSME maintains higher test performance at most intermediate stages, indicating that meta‑evolution continuously refines generation, revision, and filtering strategies.

Credit analysis demonstrates that the proportion of helpful skills rises while harmful skills are controlled, confirming that meta‑evolution writes back corrective feedback to the skill‑extraction and maintenance pipeline.

Meta‑test experiments freeze the meta‑skills learned in one HiSME run and use them to initialize a static skill‑evolution run. The “static‑from‑meta” configuration significantly outperforms “static‑from‑scratch” and approaches full HiSME performance, showing that meta‑skills are high‑quality and transferable.

Future Directions: Learning to Evolve and Automatic Harness Engineering

HiSME suggests a broader paradigm: test‑time learning need not be limited to model‑parameter updates or static experience pools. External harness components—prompts, tool routing, memory, evaluators, and skill‑library maintenance policies—can also become evolvable objects.

Two promising avenues emerge: (1) Learning to Evolve , which combines textual meta‑skills with lightweight parameter learning to let the system acquire both experience content and evolution capability; and (2) Automatic Harness Engineering , which automatically adapts external harnesses for different models and tasks, enabling agents to continuously refine their own operating procedures after deployment.

In complex agent systems, the real challenge is not merely solving a single task but continuously deciding which experiences to keep, which rules to tighten, and which failure patterns signal that the evolution algorithm itself must change. HiSME answers this by allowing agents to live more than once and by giving their learning mechanisms a chance to be learned.