Iterative Agent Skill Development: Turning Expert Knowledge into Zero‑Dependency SOPs

What is an Agent Skill

Agent Skill is a modular capability bundle that encapsulates domain knowledge as natural‑language instructions, metadata, and optional resources (scripts, templates). It functions as an "operations manual" that an AI agent can load and execute on demand.

Skill Architecture

The design follows a three‑layer progressive disclosure architecture driven by a file‑system layout:

Layer 1 – Directory/overview: lowest cost, agents only need the manual’s location.

Layer 2 – Detailed commands: loaded on demand when a specific chapter is required.

Layer 3 – Full resources: complete steps and execution scripts.

Core File Structure

SKILL.md – contains two parts:

YAML frontmatter – metadata (e.g., description) that determines when the skill should be triggered.

Markdown body – the executable SOP, recommended to use a "summary‑detail" structure (core rules first, then constraints).

references/ – supplementary documents such as templates, detailed specifications, or example code.

scripts/ – deterministic scripts (Python, Bash) that replace agent reasoning whenever possible.

Verification Mechanisms

Internal self‑check: after a skill runs, the agent validates output against a checklist.

External evaluation (eval): run realistic user prompts with and without the skill (or with an older version) and compare outputs against objective criteria or human feedback.

Suitable and Unsuitable Scenarios

Suitable: semi‑automatic repetitive processes, domain‑knowledge‑driven workflows, and tasks where the agent’s context window is limited.

Unsuitable: simple tasks that LLMs can handle directly, fully deterministic pipelines better served by code, and agents with a single, narrow responsibility.

Iterative Development Practices

Decision‑tree guidance replaces fuzzy judgment with forward constraints, making agent behavior controllable. Example snippet:

### 结果处理规则
**补全未发出消息：** 若有序事件的前序有日志、后序无日志，在报告表格中补充后序事件行，tag以外字段留空，备注标记为"消息未发出"。
**消费失败处理：** 判断某 tag 是否失败，标准为 `resultFlag = N` 且该 tag 后续无 `resultFlag = Y` 的记录。
- 若后续**有** `Y`（重试成功）→ 取**第一条**失败行，调用错误详情查询
- 若后续**无** `Y`（持续失败）→ 取**每一条**失败行，调用错误详情查询
**错误详情查询（消费失败）：**

Negative constraints with alternatives provide a concrete fallback when a pattern is forbidden. Example:

### Mocking Restrictions
**Do NOT mock:**
- `public static` fields (e.g., `@AppSwitch`‑annotated configurations) – assign values directly in `@BeforeEach` and restore originals post‑test
- POJO classes or OneLog objects – initialize simple POJOs programmatically; load complex POJOs from JSON files
- Stateless static methods (e.g., utility methods for conversion/assembly) – call real implementations directly

Internal self‑check checklist (post‑generation review):

## Post‑Generation Review
- Correct test file location and naming
- Proper mock configuration without prohibited patterns
- Complete verification of return values, state mutations, and invocations
- Consistent use of AssertJ assertion patterns
- No reflection‑based testing or private member verification
- Group similar tests into parameterized tests where appropriate
- Parameterized tests use appropriate source types and handle null values correctly

External evaluation (eval) workflow consists of four steps:

Design 2‑3 realistic user prompts that trigger the skill.

Run the prompts with and without the skill (or with an older version) and collect outputs.

Assert objective criteria (e.g., presence of field X) or gather human feedback for subjective skills.

Iterate: evaluate → modify → rerun → re‑evaluate, focusing on a small set of cases each round.

After stabilising the skill, optimise the description field by testing trigger recall precision with "should‑trigger" / "should‑not‑trigger" samples, paying special attention to near‑miss false positives and missed triggers.

Collaboration and Tooling

Two meta‑skills facilitate development (provided as plain URLs):

https://skills.sh/anthropics/skills/skill-creator – generates a new skill skeleton.

https://skills.sh/softaworks/agent-toolkit/skill-judge – evaluates a skill against quality criteria.

For multi‑person collaboration, store the skills/ directory in a version‑controlled repository (e.g., Git). Installation can be performed interactively with the skills.sh command line tool.

Comparison with Full‑Featured Agent Frameworks

Skill replaces runtime services such as vector stores, graph engines, and routers with a simple file‑system layout and text‑based decision trees, achieving zero‑dependency deployment. It is less deterministic than dedicated frameworks (e.g., LangGraph) but sufficient for most expert‑driven, semi‑automatic workflows. When 100 % determinism or highly complex stateful logic is required, a full agent framework should be used.

Conclusion

Positioning : Skill is a lightweight knowledge‑encapsulation layer for semi‑automatic, expert‑driven scenarios, not a replacement for full agent platforms.

Design principles : adopt three‑layer progressive loading, use decision trees for clear branching, pair negative constraints with concrete alternatives, and enforce both internal self‑check and external eval.

Iterative workflow : leverage skill‑creator and skill‑judge to cycle through generation, assessment, and revision, continuously improving skill quality.

Within its boundaries, Agent Skill offers a zero‑infrastructure, low‑cost path for teams to convert tacit expert experience into reusable, verifiable AI capabilities.