Why Misusing Agent Skills Is Worse Than Not Using Them (A Practical Guide)

Agent Skills are often abused by letting agents generate skill documents without real problem context; the correct approach is to extract knowledge gaps from actual tasks or encapsulate frequent actions as Skills.

Flaws in the SkillsBench paper

SkillsBench evaluated 86 tasks across 11 domains with seven agent configurations, running over 7,000 trajectories. The authors claim that well‑designed Skills raise success rates by 16.2 percentage points overall and up to 51.9 points in medical tasks, but only 4.5 points in software‑engineering tasks, with 16 tasks performing worse when Skills are used.

The paper’s “self‑generated Skills” method simply asks an agent to write a knowledge dump before solving a problem, which the author argues is equivalent to forcing the model to produce irrelevant filler text, offering no real benefit.

What Skills actually are

Skills are Markdown files with metadata that tell an agent when to use them, accompanied by scripts or reference documents. Each Skill resides in its own folder. Example structure for a GitLab CI monitoring Skill:

.claude/skills/
└── monitor-gitlab-ci/
    ├── SKILL.md
    ├── monitor_ci.sh
    └── references/
        ├── api_commands.md
        ├── log_analysis.md
        └── troubleshooting.md

SKILL.md describes the testing environment and the conditions under which the agent should act. The monitor_ci.sh script handles the actual CI polling, while the references folder stores edge‑case handling strategies.

Skills fill the agent’s “memory loss” gap

Agents are stateless; each new conversation starts with no knowledge of prior interactions. By providing a Skill that encodes project‑specific setup (e.g., test environment details), the agent avoids costly token usage and repeated trial‑and‑error when encountering large codebases or complex Docker configurations.

Automating repetitive work with Skills

Common repetitive checks—document alignment, merge‑request description consistency, issue‑code matching—can be wrapped into a single Skill (e.g., “alignment check”). Invoking the Skill once runs the entire workflow, saving time and reducing manual prompting.

Learning from failure: turning obstacles into Skills

When an agent repeatedly fails on a specific problem (e.g., a complex permission check), the missing information is extracted, distilled into a Skill, and reused. This transforms a knowledge blind spot into a reusable asset rather than a one‑off fix.

Empirical test of a practical workflow

Disagreeing with the paper’s methodology, the author reran experiments by first letting the agent attempt tasks, recording obstacles, converting those obstacles into Skills, and then re‑executing the tasks with the new Skills. The results mirrored expectations: agents equipped with practice‑derived Skills showed markedly higher success rates, especially on tasks where the paper reported degradation.

When to create a Skill

Only two scenarios merit a Skill: (1) capturing a new problem the agent solved after encountering a knowledge gap, and (2) automating a repetitive action the agent performs frequently. Starting a fresh conversation and asking the agent to “write a Skill about X” without concrete project context yields generic or incorrect output.

In summary, Skills are tools for recording experience, not for fabricating it. Let the agent solve real problems first, then codify the learned solutions as Skills for future reuse.