AWS Skill Eval Turns Agent Skills into Testable Engineering Assets

01 The Core Problem Shifts from Writing Skills to Verifying Execution

AWS defines a Skill as a reusable set of domain instructions stored in SKILL.md, containing operational steps, tool dependencies, reference knowledge, multi-step workflows, and validation constraints. This structure solves the practical problem of bloated system prompts and poor reusability across agents: agents first see a catalog of available skills, then dynamically select and load the relevant one for a given task.

Two failure modes emerge. First, wrong skill selection : an HR agent with both Benefits and PTO skills might load the PTO skill when asked about dental coverage, yet still produce a plausible answer from model knowledge. Second, correct skill selected but incomplete execution : in a PTO example, the skill requires identifying employee ID, querying balance, checking rollover policy, then submitting the request; an agent might skip the policy check and still return a seemingly correct result. Traditional final-answer-only evaluation cannot detect these gaps because a correct output does not prove the required steps were executed.

02 AWS Decomposes a Skill Execution into Three Verification Layers

The new Skill Eval introduces three distinct checks:

Skill Invoked — a deterministic assertion (no judge model) that verifies the expected skill was loaded. Example: a test case requiring pdf-table-extraction for summarizing financial tables in a PDF fails immediately if that skill does not appear.

Skill Selection Accuracy — evaluates whether the chosen skill is appropriate for the task, using the user request, available skill list, actual invocation, and context. It returns Yes/No per invocation, effectively checking the agent's routing logic.

Skill Instruction Following — reads the loaded SKILL.md, identifies required steps, and judges each step from the execution trace as Fully Followed, Mostly Followed, Partially Followed, Minimally Followed, or Not Followed. In a PDF example, the agent loaded pdf-table-extraction and ran pdftotext -layout but did not open the extracted file or locate table boundaries, yielding an Instruction Following score of only 0.5 despite perfect Skill Selection.

These layers let teams pinpoint whether a failure is due to skill not triggering, wrong skill chosen, or skill chosen but not fully executed.

03 Agent Evaluation Moves from Final Answer into Execution Traces

Traditional LLM eval focuses on input-output correctness. Agent eval added tool selection and parameter accuracy. Skill Eval inserts a new observable layer in the execution chain: user task → skill selection → skill loading → skill instruction execution → tool calls → tool parameters → task completion. Each layer can fail independently.

Implementation relies on execution trajectories. Strands Evals runs evaluations on recorded development-phase trajectories. AgentCore Evaluations reads OpenTelemetry traces directly, extracting skill invocations from sessions, traces, and spans to run skill evals on the corresponding calls. AgentCore supports on-demand, batch, and online evaluation modes, enabling ad-hoc debugging, historical batch analysis, and continuous production sampling. This brings skill issues from offline prompt testing into production observability : teams can see exactly which skill loaded, which steps executed or were skipped, and at which span the problem occurred.

04 SKILL.md Enters CI/CD — Skills Gain Software-Component Engineering Infrastructure

AWS integrates Skill Eval into deployment pipelines. In Strands Evals, developers write regression cases for critical skills (e.g., a task class must trigger a specific skill, with checks on both selection and instruction following). The evaluation report exposes report.test_passes; any failure exits the build and blocks downstream stages. AWS recommends adding deterministic SkillInvoked assertions for known-correct routing scenarios instead of relying solely on judge models.

A skill change now follows a lifecycle: modify SKILL.md, run test cases, verify invocation/selection/execution, pass gate, deploy, then continue sampling from production traces. This is where "skills move from prompt engineering to software engineering" becomes real — not because markdown becomes code, but because skills acquire engineering facilities previously reserved for software components : defined input scenarios, behavioral contracts, test cases, regression gates, deployment gates, and post-deployment trace-based continuous evaluation. The Strands Evals SDK (GitHub repository) lists Skill Evaluation as a standalone capability with CLI, experiment persistence, and versioning.

Note: SkillInvoked is deterministic, but Skill Selection Accuracy and Skill Instruction Following remain LLM-as-a-Judge evaluations, subject to model judgment variance and requiring business-specific threshold calibration. Pass thresholds differ between Strands Evals and AgentCore Evaluations and are not directly interchangeable. "Unit test" is an engineering analogy: skills gain test and release mechanisms akin to software components.

05 Competition Shifts from "Who Has Skills" to "Who Can Maintain Skills"

Strands Evals can recognize skill signals from diverse agent harnesses — Strands AgentSkills, Claude Code, Claude Agent SDK, OpenAI Agents SDK, Codex, Gemini CLI, OpenHands, Google ADK, and generic SKILL.md reads — abstracting skill invocations into a common observable object across runtimes. AgentCore's native production trace recognition currently covers Strands Agents, LangGraph Deep Agents, Google ADK, and Claude Agent SDK; other frameworks fall back to SKILL.md file detection.

As skill catalogs grow to dozens or hundreds, new challenges appear: catalog drift (overlapping skill descriptions), routing errors from novel user phrasing, and instruction-following degradation in long sessions . AWS lists these as targets for online evaluation. The problems now mirror software module scaling: regression from changes, routing interference, converting new production failure modes into regression cases, deciding which errors block releases versus serve as observations. AWS has not solved all of this, but it has supplied a critical missing piece: skills must not only be created, they must be tested .

Agent engineering has focused on giving models more tools, context, and skills. As those accumulate, the question becomes: how to prove capabilities are correctly selected and correctly executed every time, and that a change today does not silently break yesterday's working flows. The next competitive frontier is not the count of SKILL.md files, but the ability to version, test, release, observe, and continuously regress skills like code. AWS's September 22 update advances the "test" and "production evaluation" pieces of that loop.