Engineering Agent Plugins: Making Skills Verifiable, Maintainable, and Evolvable

Introduction

Agent plugins deliver a capability (Skill) that an Agent can discover and execute. A Skill is defined by a SKILL.md file that specifies trigger conditions, execution flow, and required resources. When a Skill becomes part of a long‑lived plugin and is invoked repeatedly across users, projects, and hosts, it must be treated as software rather than a one‑off prompt.

What is an Agent Plugin?

An Agent Plugin defines the delivery boundary of a capability. A typical plugin layout is:

my-plugin/</code>
<code>├── plugin.json</code>
<code>├── skills/</code>
<code>│   └── summarize/</code>
<code>│       ├── SKILL.md</code>
<code>│       ├── scripts/</code>
<code>│       └── references/</code>
<code>├── mcp.json</code>
<code>└── com.example.client/</code>
<code>    └── hooks/

The SKILL.md file is the entry point for the capability; it is not the plugin itself.

1. Spec‑Driven: Verifiable Contracts

Before implementing a change that may affect a Skill, script, template, host adaptation, or review workflow, a spec is recorded in AGENTS.md with traceable acceptance criteria (e.g., AC‑01, AC‑02, …). A spec must answer:

Which requests should trigger the Skill and which similar requests should not?

What information must be read?

Which tools may be called?

Which behaviors are prohibited?

What should happen when evidence is insufficient (stop, degrade, or hand back to a human)?

What evidence proves the implementation meets expectations?

Each acceptance criterion receives a stable identifier that links to implementation, tests, and review evidence. A comprehensive spec can decompose a reproduced issue from AC‑01 to AC‑09.

The spec‑review process ( triangulate‑spec‑review) requires at least two (often three) review Agents to examine the same context for implementation complexity, usability, and long‑term evolution. The lead Agent merges similar issues, verifies evidence, and updates documentation; other reviewers provide independent judgments without editing files.

Overly fine‑grained specs can cause AI models to perform literal string matching, reducing test stability. Specs should constrain observable behavior rather than lock specific wording.

2. Context Orchestration: Progressive Knowledge Delivery

Agents discover a Skill by reading its name and description. After selection, the full SKILL.md is loaded, and finer‑grained data is brought on demand. The recommended directory layout keeps the entry short:

my-skill/</code>
<code>├── SKILL.md    # trigger conditions, main flow, stop conditions</code>
<code>├── references/ # on‑demand judgment rules</code>
<code>├── scripts/    # deterministic logic</code>
<code>└── assets/     # templates and delivery skeleton

Three guarantees ensure reliable routing:

Discoverable : The SKILL.md entry and explicit references bring related material into the Agent’s knowledge path.

Reachable : Automated tests verify that relative links are valid and that required resources are reachable via the entry route.

Traceable : A documentation‑graph generator creates Mermaid diagrams from real Markdown references; the generated graph is validated against the current reference relationships.

When documentation moves or links break, machines detect the issue without manual syncing.

3. Deterministic Verification: Programmatic Boundaries

Boundaries that are clearly defined and decidable are handled by deterministic checks rather than by the model. Better Harness uses three layers:

Static lint checks : Verify file headers, directory layout, broken links, data formats, and permission declarations.

Unit tests : Validate scripts, parsers, and template transformations.

Contract tests : Verify that commands and entry points allow the intended inputs, outputs, error states, artifact locations, and side‑effect limits.

Example: testing the --help path ensures the command does not read the workspace, write files, wait for stdin, spawn subprocesses, or access the network. Any such side effect causes the test to fail.

python3 <skill-creator-root>/scripts/quick_validate.py <skill-dir></code>
<code>qodercli --cwd <neutral-dir> --plugin-dir <plugin-root> -p "<forward-test-prompt>"</code>
<code>qodercli plugin validate <plugin-root>

Static checks prove syntactic soundness; behavior testing is needed to confirm that the Agent selects the Skill at the right time, performs required steps, and halts when evidence is insufficient.

4. Behavior Testing: Proving Execution

Better Harness defines three scenario categories:

Positive cases that should trigger.

Negative cases that must not trigger.

Boundary cases with similar wording but different intent.

During each run four aspects are observed:

Selection : Correct use vs. accidental use.

Context : Whether the Agent reads the truly required resources.

Execution : Alignment of key steps, tools, and permissions with constraints.

Result : Whether the final artifact passes independent verification.

Each scenario is executed repeatedly to ensure stability. End‑to‑end validation uses the real Qoder CLI from a neutral directory to avoid accidental borrowing of repository configuration or undeclared dependencies.

python3 <skill-creator-root>/scripts/quick_validate.py <skill-dir></code>
<code>qodercli --cwd <neutral-dir> --plugin-dir <plugin-root> -p "<forward-test-prompt>"</code>
<code>qodercli plugin validate <plugin-root>

Behavior testing confirms that the Agent “saw” the Skill, but does not prove that the Skill improves outcomes.

5. Evidence Loop: Comparative Effectiveness

To answer whether a Skill actually improves performance, Better Harness runs an evaluation protocol with three groups under identical task, model, tool, permission, and environment conditions:

No Skill.

Current‑version Skill.

Candidate‑version Skill.

The evaluation records not only final success rate but also:

Execution of key steps.

Completeness of verification.

Time and token cost.

Any extra side effects.

A failure mode called routed‑but‑not‑applied occurs when the Skill is discovered and its SKILL.md is read, yet the required steps are never executed.

The evidence chain checks:

Existence : The Skill and its mechanisms exist.

Discoverability : A real task can find and select it.

Execution : Required key steps actually occur.

Effectiveness : Compared to not using the Skill, the result is demonstrably better.

This aligns with recent research: SkillsBench measures result differences with vs. without a Skill, while Skill Coverage checks whether the behaviors demanded by a Skill appear in the execution trace. Both result improvement and process coverage are required before self‑evolution (e.g., Trace2Skill, EvoSkill) is considered safe.

Conclusion

Engineering Agent plugins means treating the capabilities inside plugins as software assets: write verifiable specs, orchestrate knowledge delivery, guard boundaries with deterministic verification, confirm execution through behavior testing, and finally assess effect with comparative evidence. When these mechanisms work together, a plugin moves from “just usable” to “verifiable, maintainable, and sustainably evolvable”.