Decoding Anthropic’s Agent Evaluation Methodology: Challenges, Graders, and Best Practices

Anthropic’s engineering team published “Demystifying Evals for AI Agents,” which systematizes their internal evaluation practice. The article first shows a failure case where Opus 4.5 passed the test code but exploited a loophole in the strategy document to produce a better result, illustrating that static tests assuming a single correct path no longer suffice for modern agents.

Why Agent Evaluation Is Harder Than Traditional Unit Testing

Traditional unit tests involve a single input‑output call, similar to early LLM evaluations (one prompt, one response, one rule). Agents extend this chain across multiple tool calls, state changes, and reasoning steps, creating three new challenges:

Error propagation along the chain – a mistake early (e.g., reading the wrong file at step 3) corrupts later reasoning, producing a seemingly normal final output.

Multiple valid paths – the same task can be solved with different tool combinations; any correct result is acceptable.

Models find unanticipated solutions – as in the Opus 4.5 τ2‑bench ticket‑booking example, the model solved the problem in a way not foreseen by the designers.

These points imply that an agent’s correctness must be judged not only by its final text but also by its intermediate actions, state changes, and overall transcript.

Key Terminology

Task : a concrete test with inputs and success criteria.

Trial : a single attempt at a task; multiple trials are needed because model outputs are stochastic.

Grader : logic that judges a specific aspect of an agent’s performance; a task can have multiple graders.

Transcript : the full record of a trial, including outputs, tool calls, reasoning steps, and intermediate results.

Outcome : the final state of the environment after a trial (e.g., whether a ticket was actually booked).

Evaluation harness : infrastructure that dispatches tasks, runs trials concurrently, records processes, scores, and aggregates results.

Agent harness : the external system that enables a model to act as an agent, handling input processing, tool scheduling, and result return. Evaluating an agent actually evaluates the combination of harness and model.

Evaluation suite : a collection of related tasks (e.g., a set of customer‑service scenarios).

Two frequently confused pairs are Transcript vs. Outcome (the former shows what the agent did; the latter shows the final environment state) and Evaluation harness vs. Agent harness (the former runs the test; the latter runs the agent).

When to Start Evaluating

Early on, teams can rely on dog‑fooding and intuition. As the product scales, discrepancies between user feedback and internal impressions make systematic eval essential. Anthropic’s Claude Code began with internal user feedback, later adding narrow‑dimensional evals (concision, file edits) and eventually more complex over‑engineering evals. Bolt, a newer AI coding tool, added eval after real‑world usage, building a static‑analysis + browser‑agent + LLM‑judge pipeline in three months.

Early eval provides two often‑underestimated benefits: (1) specification clarification—turning ambiguous requirements into executable criteria, and (2) faster model‑upgrade cycles—teams with eval can assess a new model in days instead of weeks.

Three Types of Graders

Code‑based : string matching, unit tests, static analysis, tool‑call checks, state assertions. Strengths: fast, cheap, reproducible, debuggable. Weaknesses: brittle to legal variations, poor at subjective tasks.

Model‑based : rubric scoring, natural‑language assertions, pairwise comparisons, multi‑judge consensus. Strengths: flexible, extensible, captures nuanced judgments. Weaknesses: stochastic, more expensive, requires human calibration.

Human : expert review, crowdsourcing, sampling, A/B, multi‑rater agreement. Strengths: gold‑standard, handles highly subjective tasks. Weaknesses: slow, costly, needs skilled reviewers.

Typical composition: Code graders enforce hard criteria (e.g., test pass, correct state), Model graders assess subjective dimensions (tone, coverage, clarity), and Human graders periodically calibrate Model judges.

Task scores can be combined by weighted sum, binary all‑pass, or hybrid schemes, depending on whether partial achievement is acceptable.

Practical Tips for LLM‑as‑Judge

Provide an “Unknown” fallback: instruct the prompt to return UNKNOWN when uncertain to reduce hallucinations.

Score dimensions separately: have a judge evaluate one dimension at a time (e.g., factual accuracy, coverage, tone) for more stable results.

Design hack‑resistant graders: ensure agents must truly solve the problem to earn points; avoid graders that can be bypassed.

Periodically calibrate with human experts: after calibration, human checks can be reduced but never eliminated.

Capability Eval vs. Regression Eval

Capability eval asks what an agent can currently do. Initial pass rates should be low (e.g., ~20%) to leave room for improvement. If a new eval immediately scores 95%, the task is likely too easy.

Regression eval asks whether previously achievable behavior is still working, aiming for near‑100% pass. A drop indicates a regression.

When a capability eval reaches near‑full score, it graduates to a regression suite, shifting focus from “can it do it?” to “does it still do it reliably.” SWE‑bench Verified’s progression from 40% to over 80% exemplifies this lifecycle.

If a task never passes in 100 trials (0% pass@100), the eval itself is likely broken rather than the model.

Evaluation Differences Across Agent Types

Coding Agent : correctness is measured by test pass rates and code quality reviews. Benchmarks like SWE‑bench Verified and Terminal‑Bench follow this pattern.

Conversational Agent : the interaction is part of the evaluation. Success criteria include ticket resolution, transcript length (e.g., within 10 turns), and tone, often using LLM rubrics. Benchmarks τ‑Bench and τ2‑Bench simulate user‑agent dialogues.

Research Agent : quality is highly subjective (completeness, source reliability). Evaluation checks groundedness, coverage, and source quality, with LLM rubrics for subjective parts and periodic human calibration.

Computer‑Use Agent : interacts with GUIs; success is judged by URL changes, page states, and backend effects (e.g., order creation). Benchmarks WebArena and OSWorld illustrate these checks. Trade‑offs exist between DOM‑based interaction (fast, token‑heavy) and screenshot‑based interaction (slow, token‑light).

pass@k vs. pass^k

Because model outputs are stochastic, two metrics are used:

pass@k : probability that at least one of k attempts succeeds. Larger k yields higher values (e.g., pass@1 = 50% means a 50% chance of success on the first try).

pass^k : probability that all k attempts succeed. Larger k yields lower values (e.g., if single‑attempt success is 75%, then pass^3 ≈ 42%).

Choice depends on product needs: code‑completion uses pass@1, while a customer‑service agent that must be reliable every time uses pass^k.

Key Decisions When Building Eval from 0 to 1

Start early with real failures : 20–50 tasks derived from bug reports or support tickets are enough; small samples give strong signals.

Test both directions : evaluate that the agent triggers when it should (under‑triggering) and refrains when it shouldn’t (over‑triggering) to avoid degenerate behavior.

Isolate each trial : run trials in a clean environment to avoid state leakage; shared state can create spurious correlations or unfair advantages.

Score outcomes, not paths : avoid hard‑coding specific tool‑call sequences; instead, enforce criteria on final outputs and critical states.

Read transcripts : regularly review full trial logs to diagnose grader errors or ambiguous task descriptions.

Watch eval decay : tasks saturate, graders become outdated, and environments drift; maintain eval as a long‑term asset.

Broaden task authorship : involve product, customer‑success, and sales teams in writing tasks, as they understand “good” outcomes better than engineers.

Layered Evaluation Approach

No single method is sufficient; combining multiple layers mitigates blind spots:

Automated eval for fast, reproducible CI checks.

Production monitoring to capture real‑user behavior.

A/B testing when traffic permits.

Continuous user feedback for unexpected issues.

Manual transcript review to build intuition about failure modes.

Systematic human evaluation for gold‑standard calibration.

This “Swiss‑cheese” model ensures that the holes of one layer are covered by another.

Takeaways

Eval acts as a high‑bandwidth communication protocol between product and research, turning ambiguous goals into executable criteria.

Scoring outcomes is more robust than scoring intermediate paths, though process constraints remain necessary for safety‑critical scenarios.

Reading transcripts is essential for diagnosing grader failures and understanding why an agent succeeded or failed.

Eval itself degrades over time; treat it as a maintainable asset rather than a one‑off script.

Common Eval Frameworks (Appendix)

Harbor : container‑native platform for running large numbers of trials across clouds; used by Terminal‑Bench 2.0.

Braintrust : combines offline evaluation, production observability, and experiment tracking with built‑in factuality and relevance scorers.

LangSmith : tracing, offline/online eval, dataset management; tightly integrated with the LangChain ecosystem.

Langfuse : self‑hosted open‑source alternative suitable for data‑compliance requirements.

Arize : open‑source Phoenix tracing + eval platform; AX SaaS version adds large‑scale optimization and monitoring.

Frameworks accelerate deployment, but the ultimate value of eval depends on the quality of tasks and graders, not the platform itself.