AWS Skill Eval: Making Agent Skills Testable, Traceable, and CI-Ready

From Prompt Engineering to Software Engineering for Agent Skills

Agent Skills — reusable domain instructions stored as SKILL.md files — have become standard in agent development. They let agents dynamically load relevant procedures (expense policies, code-review rules, approval workflows) instead of stuffing everything into a system prompt. But as skill catalogs grow, two failure modes emerge:

Wrong skill selected : An HR agent with both Benefits and PTO skills might load the PTO skill when asked about dental coverage, yet still produce a plausible answer from model knowledge.

Right skill loaded, but steps skipped : A PTO skill requires (1) identify employee ID, (2) query PTO balance, (3) check rollover policy, (4) submit request. An agent that jumps from balance check to submission misses the policy validation, even if the final answer looks correct.

Traditional final-answer evaluation cannot catch these because a correct-looking output does not prove the required steps were executed.

AWS Skill Eval: Three-Layer Inspection of a Single Skill Execution

Released September 22, the new Skill Eval in Strands Evals and Amazon Bedrock AgentCore Evaluations breaks one skill invocation into three independent checks:

Skill Invoked — Deterministic assertion (no judge model) that the expected skill was loaded. Example: a test case requiring pdf-table-extraction for summarizing financial tables in a PDF fails immediately if that skill is absent.

Skill Selection Accuracy — LLM-as-a-judge evaluates whether the chosen skill matches the user task, given the available skill list and context. This isolates the agent's routing decision from downstream tool calls.

Skill Instruction Following — Reads the loaded SKILL.md, extracts required steps, and scores execution against the trace: Fully Followed, Mostly Followed, Partially Followed, Minimally Followed, Not Followed . In a PDF example, the agent loaded pdf-table-extraction and ran pdftotext -layout but never opened the extracted file or located table boundaries, yielding an Instruction Following score of 0.5 despite perfect Skill Selection.

Evaluation Moves from Final Answer into the Execution Trace

Agent evaluation has evolved: LLM eval → input/output correctness; Agent eval → tool selection, parameter accuracy, goal success. Skill Eval inserts a new observable layer between task and tool calls:

User task → Skill selection → Skill loading → Skill instruction execution → Tool calls → Tool parameters → Task completion.

Each layer can fail independently.

Strands Evals runs evaluations on recorded development-time trajectories ; AgentCore Evaluations reads OpenTelemetry traces (Session, Trace, Span) to locate skill invocations and apply Skill Eval. AgentCore supports on-demand, batch, and online modes — enabling ad-hoc debugging, historical batch analysis, and continuous production sampling.

Skills Enter the CI/CD Pipeline

Strands Evals lets developers write regression cases for critical skills: assert that a task triggers a specific skill, then verify Skill Selection and Instruction Following. The test result report.test_passes can fail the build and block deployment if any check fails. AWS recommends adding deterministic SkillInvoked assertions for known routing outcomes, reducing reliance on judge models.

A skill change now follows a lifecycle: edit SKILL.md → run test cases → verify invocation, selection, step completion → deploy → production traces continuously sampled for evaluation.

Caveat : SkillInvoked is deterministic, but Skill Selection Accuracy and Instruction Following remain LLM-as-a-Judge with inherent variance. Pass thresholds differ between Strands Evals and AgentCore Evaluations; they are an engineering analogy to unit tests, not a drop-in replacement.

Multi-Harness Observability and the Maintenance Challenge

Strands Evals can extract skill signals from diverse agent harnesses: Strands AgentSkills, Claude Code, Claude Agent SDK, OpenAI Agents SDK, Codex, Gemini CLI, OpenHands, Google ADK, and generic SKILL.md reads. AgentCore's native production trace recognition currently covers Strands Agents, LangGraph Deep Agents, Google ADK, and Claude Agent SDK; others fall back to SKILL.md file detection. The evaluation layer abstracts skill invocations into a common observable object across runtimes.

As skill counts scale to dozens or hundreds, new problems appear:

Catalog drift — overlapping skill descriptions confuse the router.

Expression gap — real user phrasing diverges from test prompts.

Instruction Following degradation — long sessions cause step dropout.

These mirror classic software module scaling issues: regression testing, release gating, observability, and continuous evaluation. AWS's update addresses the test and production evaluation gaps, shifting competition from "who has more skills" to "who can version, test, release, observe, and continuously regress skills like code."