Anthropic Adds a Full Evaluation Framework to Skill Creator

Anthropic updates Skill Creator with an evaluation framework

Anthropic has released a new version of Skill Creator that adds a complete, code‑free evaluation framework for AI skills. The update addresses the problem that most skill authors are domain experts rather than engineers and lack tools to verify skill effectiveness, correct triggering, and post‑modification improvements.

Two skill categories, different testing needs

Skills are divided into:

Capability‑enhancing skills – they enable Claude to perform tasks the base model cannot or does poorly, e.g., Anthropic's document‑creation skill that encodes specific tricks and patterns.

Workflow‑encoding skills – they record a series of steps that Claude can execute individually, but the skill strings the steps together according to team processes, such as an NDA‑review skill or a weekly‑report drafting skill from multiple MCP data sources.

The distinction matters because the former may become unnecessary as the model improves, while the latter’s value depends on how faithfully it reproduces the actual workflow.

Testing and improving skills with evaluations

The new framework lets you write evaluations that check whether Claude responds as expected to given prompts. Authors define test prompts (optionally with files) and describe what constitutes a good result; Skill Creator then reports whether the skill passes.

For example, the PDF skill previously failed to place text at exact coordinates on non‑fillable forms. An evaluation identified the issue, and a fix was released that anchors placement to extracted text coordinates.

Main uses of evaluations

Capture quality regressions : as models and surrounding infrastructure evolve, a skill that worked last month may behave differently today. Running evaluations on new model versions provides early signals before the problem impacts the team.

Know when the base model surpasses the skill : for capability‑enhancing skills, if the base model passes the evaluation without the skill, it indicates the skill’s technique has been incorporated into the model and the skill may no longer be needed.

Benchmark mode

A new benchmark mode runs standardized assessments using your evaluations after model updates or skill iterations. It tracks pass rate, latency, and token usage, and all results belong to you for local storage, dashboard integration, or CI pipelines.

Multi‑Agent support for faster, cleaner testing

Sequential evaluation can be slow and cause context contamination. Skill Creator now launches independent agents to run evaluations in parallel, each with a clean context, its own token budget, and timing metrics, resulting in faster, interference‑free results.

A comparator agent enables A/B testing: two skill versions or a skill versus no skill are run blind, and the agent decides which output is better, letting you verify whether changes truly help.

Optimizing skill trigger descriptions

Evaluations also measure output quality, but a skill must fire at the right time to be useful. Skill Creator now analyzes the current description against sample prompts and suggests edits to reduce false positives and false negatives. In Anthropic’s own document‑creation skill, five of six public skills saw improved trigger performance after using this feature.

Conclusion and outlook

As model capabilities grow, the line between “skill” and “specification” may blur. Today a SKILL.md file is essentially an implementation plan that tells Claude what to do. Over time, natural‑language descriptions of what a skill should do might suffice, with the model handling the rest. The released evaluation framework is a step toward that future, describing “what to do” and eventually becoming the skill itself.

All Skill Creator updates are now available on Claude.ai and Cowork, and Claude Code users can install the official plugin or download it from the GitHub repository.