AWS Skill Eval: Turning Agent Skills into Testable Engineering Components

Agent Skills have become a standard pattern: teams encode code-review rules, document-processing steps, or approval workflows into SKILL.md files that agents load dynamically instead of stuffing everything into the system prompt. This improves reuse and maintainability, but introduces a new problem — how do engineers know the agent actually used the right skill and followed every step?

Two Failure Modes

AWS identifies two failure modes. First, wrong skill selection : an HR agent with both Benefits and PTO skills might load the PTO skill when asked about dental coverage, yet still produce a plausible answer from its own knowledge. Second, correct skill but incomplete execution : the PTO skill requires identifying the employee ID, checking balance, verifying the latest rollover policy, then submitting the request; an agent that skips the policy check may still return a seemingly correct result.

Traditional final-answer-only evaluation cannot catch these because a correct-looking output does not prove the required steps were executed.

Three-Layer Skill Evaluation

AWS adds three evaluation layers in Strands Evals and Amazon Bedrock AgentCore Evaluations:

Skill Invoked — deterministic check (no judge model) that the expected skill was loaded for a given input. Example: a test case requires pdf-table-extraction when the user asks to summarize financial tables in a PDF. If the skill is absent, the test fails immediately.

Skill Selection Accuracy — judges whether the chosen skill is appropriate for the task, using the user request, available skill list, actual invocation, and context. It evaluates the agent's routing decision.

Skill Instruction Following — reads the loaded SKILL.md, extracts required steps, and compares them against the execution trace. Results are graded on a five-point scale: Fully Followed, Mostly Followed, Partially Followed, Minimally Followed, Not Followed. In a PDF example, the agent loaded pdf-table-extraction and ran pdftotext -layout but never opened the extracted file or located table boundaries, so Instruction Following scored only 0.5 despite perfect Skill Selection.

From Final Answer to Trace-Based Evaluation

Skill Eval inserts a new observation layer into the execution chain: user task → skill selection → skill loading → skill instruction execution → tool calls → tool parameters → task completion. Each layer can fail independently. Strands Evals runs evaluations on recorded development trajectories; AgentCore Evaluations reads OpenTelemetry traces (Session, Trace, Span) to locate skill invocations and run evaluations on them. AgentCore supports on-demand, batch, and online (continuous sampling from production traffic) evaluation modes, bringing skill observability into production.

Skill Evaluation in CI/CD

AWS integrates Skill Eval into deployment pipelines. In Strands Evals, developers write regression cases for critical skills (e.g., a task must trigger a specific skill, and both Selection and Instruction Following must pass). The build fails if report.test_passes is false, blocking release — "like a failed unit test." For known routing outcomes, AWS recommends deterministic SkillInvoked assertions instead of relying solely on LLM-as-a-Judge. A skill change now follows a lifecycle: edit SKILL.md, run test cases, verify invocation/selection/execution, deploy, then continue sampling from production traces.

This engineering infrastructure — defined input scenarios, behavioral contracts, test cases, regression gates, deployment gates, and post-deployment trace-based evaluation — marks the shift from prompt engineering to software engineering for skills. However, Skill Selection Accuracy and Instruction Following remain LLM-as-a-Judge evaluations with inherent variance; thresholds need calibration per business data, and pass/fail criteria differ between Strands Evals and AgentCore Evaluations.

Cross-Framework Skill Observability

Strands Evals can extract skill signals from multiple agent harnesses: Strands AgentSkills, Claude Code, Claude Agent SDK, OpenAI Agents SDK, Codex, Gemini CLI, OpenHands, Google ADK, and generic SKILL.md reads. This does not mean these frameworks share a runtime; rather, the evaluation layer abstracts skill invocations into a common observable object. AgentCore's native production trace recognition currently covers Strands Agents, LangGraph Deep Agents, Google ADK, and Claude Agent SDK; others fall back to SKILL.md file detection.

Scaling Challenges: Catalog Drift and Instruction Following Degradation

As skill catalogs grow to dozens or hundreds, new issues emerge: catalog drift (overlapping skill descriptions), routing errors from novel user phrasing, and Instruction Following degradation in long sessions . AWS lists these as targets for online evaluation. The problems mirror software module scaling: regression from changes, routing interference, converting production failure modes into new regression cases, and deciding which errors block releases versus serve as observability metrics.

The article concludes that the next competitive frontier for Agent Skills is not quantity but maintainability — version control, testing, release, observation, and continuous regression — and AWS's September 22 update advances the "test" and "production evaluation" pieces of that lifecycle.