Writing Effective Agent Skills: Human‑Crafted Boost +16.2 pp, AI‑Generated Drop –1.3 pp

Adding a "skill package" (Skill) to an AI Agent can be beneficial, but only when designed correctly. SkillsBench, the first systematic benchmark for evaluating Agent Skills, covers 84 tasks in 11 domains with 7,308 test trajectories, testing Claude Code, Codex, and Gemini CLI.

A Harsh Comparison

Core data:

Human‑crafted Skill : average improvement +16.2 pp

Model‑generated Skill : average change ‑1.3 pp (often detrimental)

The contrast is stark: well‑written Skills boost performance dramatically, whereas letting the AI write its own Skills yields almost no benefit.

Conclusion 01: Don’t Rely Solely on AI‑Generated Skills

SkillsBench finds that models cannot reliably produce the procedural knowledge they would benefit from. The typical failure mode is generating vague guidance such as “use pandas to process data” without concrete API calls.

However, AI can assist when provided with good context: it can solidify operation paths, output styles, and formats, but full autonomy in Skill creation is not yet mature.

Best combo: Human defines framework and domain, AI fills details.

Conclusion 02: Overly Comprehensive Skills Harm Performance

Focused Skills (2‑3 modules): +18.6 pp

Large collection of files: +5.9 pp

Encyclopedic, all‑covering Skill: ‑2.9 pp (harmful)

Agents struggle to extract actionable guidance from overly verbose Skills; the context becomes saturated without clear direction. Write Skills like a runbook, not a wiki.

Conclusion 03: More Loaded Skills Can Make the Agent Dumber

Although Skills are loaded on demand, they consume context. Performance peaks with 2‑3 focused Skills (+18.6 pp) and drops to +5.9 pp when more than four are loaded, as competing instructions clash.

Conclusion 04: Test Every Skill in Each Harness

Different agents and models have varying capabilities. For example, Codex includes a drawing tool that Claude Code lacks, and Claude is multimodal while some models cannot process images. These differences require empirical testing and compatibility notes, as the same Skill may behave differently across environments.

Conclusion 05: Target the Model’s Weakest Domains

Healthcare: +51.9 pp

Manufacturing: +41.9 pp

Cybersecurity: +23.2 pp

Finance: +15.1 pp

Software Engineering: +4.5 pp

The pattern is clear: Skills help most where the model is least familiar. In software engineering, models are already saturated with code, so Skill impact is minimal.

Conclusion 06: Test or You’ll Never Know If a Skill Works

Method: select a set of tasks, run them twice—once with the Skill loaded and once without—keeping all other variables identical, then compare scores.

Even human‑crafted Skills can backfire on 16 of 84 tasks, especially where the model is already strong, due to conflicting guidance or unnecessary complexity. Pairwise testing is essential, and when quantitative results are ambiguous, expert judgment is required.

Final Takeaway

Human design → AI‑assisted refinement → Empirical validation → Iterative improvement

Skills are effective, but their impact depends heavily on domain relevance and design quality. Automated generation alone is insufficient; keep Skills concise, test rigorously, and iterate based on results.