How Real Feedback Drives Continuous Skill Evolution for AI Agents

Skill's Three‑Layer Structure

In an Agent system a Skill is a versioned capability package stored in a directory whose entry point is SKILL.md. The package consists of three layers:

Routing layer : name, description, and path, used to decide whether the Skill matches the current task.

Instruction layer : the body of SKILL.md, describing the workflow, judgment criteria, tool strategy, and output constraints.

Resource layer : sub‑folders such as references/, assets/, scripts/ that hold detailed documents, examples, templates, or executable code.

This structure enables progressive loading: the Agent first reads only the routing information, then loads the full instruction file when a task matches, and finally pulls additional resources if needed.

Self‑Evolution Through Real Feedback

Each layer can be updated based on actual task feedback. The routing layer can be refined to improve trigger boundaries, the instruction layer can be revised to stabilise workflows, and the resource layer can be enriched with new examples or scripts. Every change must answer three questions: which layer was changed, what problem it solves, and what evidence proves the improvement.

Example: Travel‑Planner Skill

Initial version travel-planner-skill v1.0:

---
name: travel-planner
description: Plan a multi‑day travel itinerary based on destination, dates, and user preferences.
---
## Workflow
1. Collect destination, dates, budget, and travel preferences.
2. Search for attractions, restaurants, and transportation options.
3. Arrange attractions by day.
4. Generate a day‑by‑day itinerary.
## Output
Use one section per day.
Include attractions, food recommendations, and transportation notes

Generated itinerary (Day 1‑3) satisfied the basic request but ignored the user’s “relaxed pace” preference and placed distant attractions together on Day 3. The user replied, “Day 3 is too rushed, I don’t want frequent transfers, and restaurants should be on the way.”

The system records the whole interaction trace, abstracts the feedback into stable rules, and updates the Skill accordingly.

Updating the Instruction Layer

First iteration modifies the workflow:

2. Search for attractions, restaurants, and transportation options.
3. Group candidate places by geographic area before assigning them to days.
4. Estimate transit time between major stops.
5. Match each day's density to the user's pace preference:
   - relaxed: 2‑3 major stops per day, with buffer time
   - standard: 3‑4 major stops per day
   - packed: 4‑6 major stops per day

This change adds geographic clustering, transit‑time estimation, and pace‑based density control, turning the plan from a simple list into an “area‑and‑pace‑organized” itinerary.

Adding Quality Checks

## Quality checks
Before finalizing the itinerary, check:
- Whether each day stays mostly within one geographic area.
- Whether transit time between major stops is reasonable.
- Whether the number of major stops matches the user's pace preference.
- Whether meals and rest breaks are placed near the route.

These checks ensure route coherence, reasonable travel time, pace fit, and proximity of meals before the Agent outputs the final plan.

Extending the Resource Layer

When users frequently mention “children, seniors, stroller, or limited walking ability”, a new reference file is added:

references/family-travel-constraints.md

It contains constraints such as reducing transfers, limiting walking distance, scheduling naps, and preferring easily reachable attractions. The main SKILL.md now includes a single trigger line:

If the user mentions children, seniors, stroller, or limited walking ability, read references/family-travel-constraints.md before finalizing the itinerary.

This keeps the core Skill lightweight while loading detailed constraints only when needed.

Periodic Skill Compaction

After many iterations the Skill becomes verbose. Compaction scans for duplicate or rarely‑triggered rules, moves stable constraints to the resource layer, and merges similar rules into higher‑level principles. Example raw rules:

- Do not place far‑apart attractions on the same day
- Limit the number of attractions per day
- Leave rest time for relaxed trips
- Keep restaurants near the day's route
- Reduce transfers for family trips

After compression they become a single quality standard:

Prioritize route coherence and pace fit: keep each day geographically coherent, limit major stops according to pace, place meals and breaks near the route, and reduce transfers for family trips.

Validation Before Release

When a candidate Skill version is generated, the system runs a batch of historical travel‑planning tasks with both the old (v1.0) and new (v1.1) versions. It compares metrics such as number of cross‑area moves, average transit time, explicit satisfaction of user preferences, and overall itinerary density. If v1.1 outperforms v1.0, it is released; otherwise the change is rejected and logged as negative feedback for the next cycle.

Conclusion

The example demonstrates that Skill self‑evolution focuses on small, verifiable updates: user feedback becomes evidence, evidence is abstracted into rules, rules are written to the appropriate Skill layer, and validation decides whether the new version is deployed. Over time, accumulated rules are compacted, merged, or moved to the resource layer, keeping the Skill clear, lightweight, and maintainable.

In the long run, an Agent’s capabilities evolve from single‑prompt adjustments to continuous Skill operations. General reasoning abilities, tool integration, and reusable Skills together form a version‑controlled, evaluable, rollback‑able, and reusable capability system where real‑world feedback continuously enriches routing, instruction, and resource layers.