Iterative Development of Agent Skills: A Hands‑On Guide

What is an Agent Skill

Agent Skill is a modular capability bundle that encapsulates domain knowledge as natural‑language instructions, metadata, and optional resources (scripts, templates). It acts as an "operations manual" for an AI agent, allowing the agent to load and execute the skill on demand without runtime services.

Skill Architecture

The design follows a three‑layer progressive disclosure model:

Layer 1 – Directory/overview (lowest cost; the agent only needs to know the skill’s location).

Layer 2 – Detailed commands (loaded on demand when a specific chapter is needed).

Layer 3 – Full resources (complete steps and tool scripts).

Skills are stored in a file‑system hierarchy. The core file SKILL.md contains two parts:

YAML front‑matter : metadata that the agent reads to decide whether to trigger the skill.

Markdown body : the execution SOP, recommended to use a "summary‑details" structure.

Supporting directories include references/ for supplemental docs and scripts/ for deterministic Python/Bash scripts that should replace agent reasoning whenever possible.

Suitable and Unsuitable Scenarios

Well‑suited: semi‑automated repetitive processes, domain‑knowledge‑driven workflows where LLM generalisation is insufficient, and contexts with limited window size.

Not suitable: simple tasks that LLMs can handle directly, fully deterministic pipelines better expressed as code, or agents with a single, narrow responsibility.

Iterative Development Practices

1. Use decision trees instead of vague judgments. Decision trees provide forward constraints that make the agent’s behaviour controllable. Example snippet:

### Result handling rules
**Complete missing messages:** If a preceding event has logs but the following event has none, add a row for the missing event in the report, leave non‑tag fields empty, and mark the note as "Message not sent".
**Failure handling:** Identify a tag as failed when <code>resultFlag = N</code> and there is no subsequent <code>resultFlag = Y</code>. If a later <code>Y</code> exists → take the first failed line; otherwise → take every failed line.
**Error detail query (failure):** …

2. Pair negative constraints with explicit alternatives. When a skill forbids a pattern, provide a concrete fallback. Example from a unit‑test skill shows a "Do NOT mock" list followed by the recommended approach.

### Mocking Restrictions
**Do NOT mock:**
- <code>public static</code> fields (e.g., <code>@AppSwitch</code> configs) – assign values directly in <code>@BeforeEach</code> and restore after tests.
- POJO classes or OneLog objects – initialise simple POJOs programmatically; load complex POJOs from JSON files.
- Stateless static methods – call real implementations directly.

3. Internal self‑check mechanism. After a skill runs, the agent validates the output against a checklist. Example checklist fragment:

## Post‑Generation Review
- Correct test file location and naming
- Proper mock configuration without prohibited patterns
- Complete verification of return values, state mutations, and invocations
- Consistent use of AssertJ assertion patterns
- No reflection‑based testing or private‑member verification
- Group similar tests into parameterised tests where appropriate
- Parameterised tests handle null values correctly

4. External evaluation (eval). Run realistic inputs with and without the skill, compare outputs, and iterate. The eval loop consists of:

Design 2‑3 realistic prompts that should trigger the skill.

Run both the skill‑enabled and skill‑disabled versions.

Assert objective criteria (e.g., presence of field X) or collect human feedback.

Iterate: evaluate → modify → rerun → re‑evaluate, focusing on a small set of cases each round.

Optimise the description field’s trigger rate by testing recall precision on "should‑trigger" and "should‑not‑trigger" samples.

Collaboration and Tooling

Two meta‑skills are recommended to bootstrap development:

skill‑creator – generates a new skill skeleton. URL: https://skills.sh/anthropics/skills/skill-creator

skill‑judge – evaluates a skill against the criteria above. URL: https://skills.sh/softaworks/agent-toolkit/skill-judge

For multi‑person collaboration, store the skills/ directory in a Git repository and install interactively with the provided command‑line tool.

Comparison with Dedicated Agent Frameworks

Skill replaces runtime services (vector stores, graph engines, routers) with a file‑system layout and natural‑language decision trees, achieving zero‑dependency deployment. This yields a lighter weight but slightly weaker determinism compared to specialised frameworks such as ReAct or LangGraph. When 100 % determinism or highly complex stateful logic is required, a dedicated framework remains preferable.

Core Technical Details

The standard Skill directory structure includes: SKILL.md – YAML front‑matter (e.g., description field) and Markdown body. references/ – supplemental documents, templates, example code. scripts/ – deterministic Python/Bash scripts that replace agent reasoning.

Progressive disclosure layers:

Layer 1: directory/overview – minimal cost, the agent only needs the path.

Layer 2: detailed commands – loaded on demand for specific chapters.

Layer 3: full resources – complete steps and tool scripts.

Decision trees are highlighted as high‑quality indicators in the skill‑judge evaluation (green flags for knowledge delta and usability). They encode expert judgment explicitly, reducing reliance on vague language.

Green flags (indicators of high knowledge delta): Decision trees for non‑obvious choices ("when X fails, try Y because Z"). D8 Usability check: Decision trees – for multi‑path scenarios, is there clear guidance on which path to take?

Negative constraints must be paired with concrete alternatives to enforce strong guidance. The skill‑judge metric D1 (knowledge increment) and D8 (usability) both reward this pattern.

Self‑check mechanisms enforce post‑execution validation, while external eval provides dynamic verification during development. The four eval steps are:

Design realistic test prompts.

Run skill‑enabled and skill‑disabled versions.

Assert objective criteria or collect human feedback.

Iterate based on observed gaps.

Optimising the description field improves trigger precision by testing recall on "should‑trigger" and "should‑not‑trigger" samples.

Conclusion

Agent Skill is a lightweight, domain‑knowledge wrapper for semi‑automated, expert‑driven scenarios.

Adopt the three‑layer progressive disclosure architecture, use decision trees, pair negative constraints with alternatives, and enforce both internal self‑check and external eval.

Iterate rapidly with skill‑creator and skill‑judge, refining the skill through a generate‑evaluate‑revise loop.