Skip to content
Five Counterintuitive Rules for Writing Effective AI Skills, Backed by Research

Five Counterintuitive Rules for Writing Effective AI Skills, Backed by Research

Three recent papers establish the first evidence-based handbook for writing AI agent skills: write skills yourself, keep them focused (2-3 files), load selectively, test per runtime framework, and target domains where model pretraining is weak.

ThinkInAI Community
ThinkInAI Community
ThinkInAI Community

What Counts as a Skill?

A skill is a packaged folder of domain knowledge that an agent loads on demand, containing instructions, vocabulary, and sometimes scripts and resources. It sits on a spectrum from prompts to tools to a new concept called "recipes" (from Roland Gavrilescu of Introspection), which add evaluations, judges, and feedback loops. The SkillsBench paper (arXiv:2602.12670, Feb 2026) formally defines skills as modular, reusable, carrying procedural guidance, and portable across models — though portability proves tricky. System prompts, RAG retrieval, and tool descriptions are not skills. The authors analogize: model = CPU, runtime = OS, skill = application.

1. Don't Let the Model Write Its Own Skills

SkillsBench tested three conditions per task: no skill, expert-curated skill, and model-generated skill. Curated skills significantly improved pass rates; self-generated skills yielded zero gain, averaging 1.3 percentage points below the no-skill baseline. Failure modes: models write vague procedures (e.g., "use pandas for data processing" without concrete API patterns) or fail to recognize the need for specialized knowledge. The SkillComposer evolution paper (arXiv:2606.06079, June 2026) found that uncontrolled skill extraction from prior logs degraded coding performance below the no-skill baseline. Paul Bakaus (AIEWF workshop) confirms: expert vocabulary is the skill's value, and models cannot supply it themselves. Gavrilescu notes you cannot simply stuff knowledge into a Markdown file unless a human extracts it first.

2. Writing a Comprehensive Skill Is Not the Right Approach

SkillsBench found that focused skills of two to three files outperformed larger bundles and exhaustive documentation. Comprehensive skills that try to record everything actually pulled pass rates below the no-skill baseline. "Detailed but bounded" beats "encyclopedic" every time. Bakaus describes this as finding the "precise layer of control": not everything belongs at the skill abstraction layer; direct manipulation is faster for tweaks, open-ended prompting still useful for exploration. Skills help when they encode what the model repeatedly gets wrong, not when they duplicate the manual. Every added section competes for the model's attention, and beyond a surprisingly low threshold, marginal content starts to hurt.

3. Loading More Skills Makes the Agent Worse

The Generative Skill Composition paper (also called SkillComposer, arXiv:2606.32025, June 30 2026) injected a full 196-skill library into context and measured a 16-point drop on coding tasks versus a small relevant subset, while consuming 23% more input tokens. Context bloat fails on both accuracy and cost. Selective loading works far better: their composer is a tiny 3.9M-parameter model that outputs an ordered skill ID list; on GPT-5.2-Codex it lifted pass rates by 23.1 points over no-skill baseline, nearing an "oracle" upper bound. Bakaus independently arrived at the same architecture inside Impeccable (https://github.com/pbakaus/impeccable), routing tasks to relevant instructions like a mixture-of-experts. Some teams route with a dedicated model outside the skill, others inside; both exist because skill efficacy depends on the mechanism that decides when to load them.

4. Test in Every Runtime Framework You Claim to Support

Skills are portable in format (everyone accepts SKILL.md) but not in behavior. SkillsBench evaluated the same skills across multiple runtimes and found gains varying from 4.1 to 25.7 points depending on the model-runtime pairing. Claude Code showed the highest skill utilization; Codex CLI often ignored skills entirely, acknowledging the content but implementing its own plan. Bakaus notes Codex and Claude handle sub-agents and permissions differently, so a skill targeting Claude Code, Cursor, GitHub Copilot, and Codex cannot assume identical capabilities. Impeccable ships different compiled versions per target runtime (https://github.com/pbakaus/impeccable), adding extra rules to forbid known bad habits of specific models. Portability is not a property declared in a compatibility field; it is engineered per target, and the runtime itself is as much part of the skill's behavior as the skill file.

5. Aim Skills at the Model's Weakest Spots

SkillsBench measured skill gains by domain: from 4.5 points in software engineering to 51.9 points in healthcare, with manufacturing close behind. Skills help least in software because pretraining already covers it heavily; they help most where coverage is thin. The authors crawled the public skill ecosystem and found 38% of all skills are for software development — the community builds skills where they help least. The arbitrage opportunity lies in domains the model doesn't know: your company's internal conventions, regulated industries, specialized workflows, anywhere pretraining coverage is shallow. A generic coding workflow skill is a crowded trade with tiny edge; a skill encoding how your compliance team reviews vendor contracts is not.

You Can't Know If Your Skill Works Without Testing

Every result above comes from the same method you can run yourself (https://github.com/benchflow-ai/skillsbench): take a task suite, run twice (with skill and without), keep all else equal, score both runs identically, compare the difference. Automated checks (scripts verifying output, returning pass/fail without human judgment) make the comparison trustworthy and repeatable. This matters because skills can be harmful: in SkillsBench, 16 of 84 tasks scored lower with the skill loaded. You cannot spot this by eyeballing outputs — skilled outputs often look more professional even when they fail more often. Only the with-vs-without comparison catches it. The SkillComposer evolution paper turns this into a quality gate: accept a skill change only if it measurably improves pass rates on the task suite. Keep what measurably helps, discard what doesn't. This turns skill maintenance from vibe into a loop — exactly the loop Gavrilescu's "recipe" concept describes: wrap evaluations and feedback signals as first-class components around the skill. Tools like Arize AX can help build such evaluations.

Remaining Gaps

How to score domains without automated checks? "Is the design bolder?" has no pass/fail. How to measure the failure mode Bakaus raised at AIEWF: if everyone uses the same skill for frontend design, everything eventually looks the same? A skill that collapses output diversity across users is failing even if each individual output scores well. Both need evaluation methods the field hasn't standardized yet.

The Folk-Wisdom Era Is Over

Five months ago, writing a good skill meant apprenticing to scattered workshop talks. Now we have a checklist backed by tens of thousands of experiments: write it yourself, keep it small, don't write too many, test per runtime, target model weaknesses, and verify every change with evals. The people who did the measurements have done the hard part. You have the data; go make it work.

Image
Image
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

prompt engineeringskill evaluationAI agent developmentagent engineeringAI skillsSkillsBenchruntime portabilitySkillComposer
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.