Claude Skill-Creator Gets Major Update: Add Unit Tests to Your Agent Skills

Why Testing Agent Skills Matters

Developers often spend hours building an Agent Skill, only to face uncertainty when models update, when minor tweaks are made, or when new Skills interact unpredictably. Most Skill authors are domain experts rather than engineers, lacking tools to confirm that Skills behave as intended.

Skill Types and Their Testing Focus

Anthropic defines two categories:

Capability‑enhancing Skills : compensate for model limitations (e.g., precise PDF form handling without field cues).

Preference‑encoding Skills : encode team‑specific workflows (e.g., NDA review checklists or weekly report templates).

The testing emphasis differs: the former monitors whether the baseline model can already perform the task, while the latter verifies strict adherence to prescribed processes.

New Feature: Unit‑Test‑Style Evals

Evals – Skill "Unit Tests"

The core addition is support for evals. Users supply a set of test prompts (with required files) and describe the expected correct output. The skill‑creator runs the Skill automatically and checks compliance.

Example: a PDF Skill previously failed to locate text in forms lacking fillable fields. An eval captured this failure scenario, prompting the team to adjust the logic to anchor text to extracted coordinates, which resolved the issue.

/plugin marketplace add anthropics/skills

Two primary benefits:

Catch quality regressions : as models and infrastructure evolve, evals provide early signals when a previously stable Skill degrades.

Detect "obsolete" Skills : for capability‑enhancing Skills, if the baseline model passes the eval without the Skill, the Skill has effectively been internalised and may be retired.

Benchmark Mode – Continuous Performance Dashboard

While a single eval run yields a snapshot, Benchmark mode aggregates multiple eval runs over time, reporting pass rate, execution time, and token consumption. Users can run benchmarks after each model update or before/after Skill modifications to track trends.

Results can be stored locally, fed into a dashboard, or integrated into CI/CD pipelines, treating Skills like software components under continuous integration.

Parallel Agents + Blind Comparison

Sequential eval execution is slow and can suffer context leakage. The updated framework runs evals in parallel agents, each in an isolated context with independent token accounting.

Blind comparison involves a third‑party Comparator Agent that evaluates two Skill versions (A vs. B or Skill vs. no Skill) without knowing which is which, eliminating subjective bias.

https://github.com/anthropics/claude-plugins-official/tree/main/plugins/skill-creator

Trigger Description Optimization

As the Skill library grows, precise trigger descriptions become critical. The new skill‑creator analyses current descriptions against historical trigger samples and suggests refinements, reducing false‑positive and false‑negative activations.

Anthropic tested this on six public documents, achieving a noticeable boost in trigger accuracy for five of them.

Looking Ahead

Anthropic notes that today’s SKILL.md acts as an "implementation plan" (how). As model capabilities increase, future Skills may only need a "goal description" (what), with the model deriving the implementation path itself. The evals framework, which specifies expected outcomes rather than procedural steps, aligns with this direction.

Getting Started

The updates are available on Claude.ai and Claude Code.

Claude Code users can install the plugin via the marketplace command above or clone the repository directly:

https://github.com/anthropics/skills/tree/main/skills/skill-creator

Two Skill types with distinct testing priorities

Evals for unit‑test‑style validation

Benchmark mode for pass rate, latency, and token metrics

Parallel execution and blind Comparator Agent for objective comparison

Trigger description tuning to improve activation precision

Future shift toward outcome‑only Skill specifications