Unlocking Anthropic’s Skill‑Creator: New Evaluation, Benchmarking, and Parallel Testing Features

During a recent live stream the author discovered that Anthropic’s Skills repository had been updated, introducing a major upgrade to the Skill‑creator – the core tool for generating and managing Skills that power agents.

New Capabilities Added

The updated Skill‑creator now includes four brand‑new abilities:

An evaluation system that automatically tells you whether a Skill works as intended.

A benchmark suite that quantifies pass rate, latency, and token usage.

Parallel multi‑agent testing with isolated environments, supporting A/B blind evaluation without cross‑contamination.

Description optimization that automatically refines Skill triggers, removing spurious activations.

Why Evaluation Matters

Previously, generated Skills were a black box – users could not tell if a Skill’s quality or trigger logic was appropriate. The new evaluation system fills this gap, providing concrete metrics and a systematic way to improve Skills.

Updating to the Latest Version

To upgrade, simply send the following command to any Anthropic‑compatible agent (Claude code, OpenClaw, OpenCode, etc.):

https://github.com/anthropics/skills/tree/main/skills/skill-creator，这个skills更新了，帮我更新到最新版本

The agent will fetch the latest code and apply the update automatically.

Demo: Turning a Video Link into a Bilingual Transcript

The author created a new Skill that, given a video URL, downloads the video (using a previously built yt‑dlp Skill) and then generates a bilingual (original language + Chinese) transcript. The prompt used was:

我想创建一个skill，我希望能够实现我给了一个视频链接，它能够把文字版的讲稿发给我，如果是别的语言，最好是把原语言版和中文版的讲稿文档给我。

After a few minutes the Skill produced a clean, well‑formatted transcript, which could then be further refined via the description‑optimization loop.

Evaluation Workflow

The evaluation process consists of three stages:

Generating two sets of 10 queries each – one set that should trigger the Skill, another that should not.

Running each query in a clean sandbox, recording success/failure, token usage, and latency.

Iteratively refining the Skill description based on the results (up to five optimization rounds, each taking ~10‑20 minutes).

Results are displayed in a matrix where green checkmarks indicate successful triggers and red crosses indicate failures. The system also splits data into a 60 % training set and a 40 % test set to avoid over‑fitting.

Quantitative Impact

In Anthropic’s internal tests on six document‑processing Skills, five showed a measurable increase in trigger accuracy. For a PDF‑processing Skill, the pass rate rose from a baseline 9 % to 100 %, while token consumption increased from ~1.7 k to ~4 k per run – a trade‑off deemed worthwhile given the quality gain.

Types of Skills and Evaluation Focus

Skills fall into two categories:

Capability‑enhancement Skills : teach Claude new abilities (e.g., front‑end design, document generation).

Encoding‑preference Skills : enforce a specific workflow or format (e.g., meeting‑notes summarizer, weekly‑report generator).

Evaluation differs accordingly: capability‑enhancement Skills are compared with and without the Skill (A/B testing) to decide if the Skill remains useful; encoding‑preference Skills are checked for strict adherence to the prescribed steps.

Conclusion

The revamped Skill‑creator brings software‑engineering rigor—testing, benchmarking, iterative improvement—to the AI‑agent ecosystem, making Skills far less of a black box and dramatically boosting their reliability and usefulness. Users are strongly encouraged to update and re‑evaluate all existing Skills.