No Evals, No Production Agents: The 4-Layer Evaluation Framework

Why Traditional QA Fails for Agents

Traditional software testing rests on three assumptions: determinism (same input → same output), enumerability (all paths can be covered), and assertability (correct results are precisely definable). Agents break all three: the same user request can take different execution paths, paths cannot be enumerated, and correctness spans six dimensions—result, path, tool usage, permissions, cost, and risk. LangChain's 2026 survey shows 89% of organizations have agent observability and 94% have full traces, yet only 52% run offline evaluations and 37% run online evaluations (LangChain, AI Observability in the Agent Development Lifecycle, 2026). Observability ≠ evaluation ≠ improvement.

Four-Layer Eval Model

Layer 1: Model Eval — Foundation Capabilities

Assesses raw model abilities (reasoning, instruction following, coding, multilingual, long-context) via standardized benchmarks (MMLU, GSM8K, HumanEval, MATH). Limitation: benchmark scores are necessary but not sufficient for real business scenarios.

Layer 2: Component Eval — Isolating Failures

Evaluates each component independently: RAG (retrieval accuracy, recall, ranking, context completeness), Tool (description clarity, parameter schema, error handling), Skill (execution quality, trigger accuracy), MCP (external service stability, schema conformance), Router (routing correctness). Core value: when overall agent performance drops, component eval pinpoints whether the model weakened, RAG retrieval degraded, or a tool interface changed.

Layer 3: Trajectory Eval — Process Over Outcome

Unique to agents. Evaluates the full execution chain: tool selection, step completeness, duplicate calls, invalid loops, permission violations. AgentTrace (ArXiv 2602.10133, 2026) proposes a three-facet observation model—operational (tool calls, latency), cognitive (reasoning steps, decision logic), and contextual (input data, environment state)—unified in a Trace Envelope. Key finding: outcome-only evaluation catches most explicit errors but misses over half of implicit errors (correct-looking results with flawed processes).

Layer 4: Outcome Eval — Business Impact

Measures whether the user's problem was truly solved, using business metrics: customer satisfaction, first-contact resolution, human escalation rate, conversion rate. Hardest to automate due to subjectivity.

Three Dataset Types as Layered Defense

Golden Set — High-Frequency Standard Tasks

100–500 cases covering top 80% of traffic. Clear tasks, clear expected results, relatively fixed paths. Used for regression on every model/prompt/tool/workflow change. Built from production logs with human review.

Edge Case Set — Low-Frequency Complex Scenarios

50–200 cases: multi-intent requests, ambiguous inputs, missing data, tool timeouts, cross-system joins. Exposes degradation in complex, multi-step scenarios where agents often fail despite passing Golden Set.

Adversarial Set — Security & Permission Boundaries

30–100 cases: prompt injection, privilege escalation, social engineering, role confusion, data exfiltration. Pass criteria stricter than Golden Set: must refuse unauthorized requests, resist injection, not leak sensitive data.

Three Evaluation Methods — Combined, Not Exclusive

Rule-based Eval

Deterministic, repeatable, low-cost, fast. Checks numeric values, JSON schema, tool call presence/absence, permission lists. Covers ~30–40% of evaluation needs; cannot assess semantic quality or user experience.

LLM-as-Judge

Uses a stronger model to evaluate semantic correctness, completeness, tone, multi-answer equivalence. Limitations: judge misclassification, positional bias, verbosity bias, judge model drift. OpenAI recommends layered judging: judge final output for semantics, step-rubric for intermediate steps, rule-based for tool calls (OpenAI Agent Evals, 2026).

Human Eval

Gold standard for high-value, high-risk, novel, or judge-uncertain cases. Expensive, slow, non-repeatable. Used to calibrate judge models periodically.

Recommended Combinations

Golden Set + Rule-based = fast regression on every change

Edge Case Set + LLM-as-Judge = deep evaluation for implicit regressions

Adversarial Set + Rule-based (permissions) + LLM-as-Judge (behavior) = security verification

Production Sampling + Human Eval = quality backstop, continuous calibration

Rule-based where possible; judge for the rest; human to calibrate judges.

Seven Core Metrics

Task Success Rate : % of eval cases where agent correctly solves user problem. Target ≥85%.

Tool Selection Accuracy : % of required-tool scenarios where agent picks the right tool. Drop signals tool description or router issues.

Trajectory Quality : 0–1 score (LLM-as-Judge or step-rubric) for path efficiency—no unnecessary steps, duplicates, or suboptimal routes. Can degrade while success rate holds, signaling early model/prompt drift.

Critical Error Rate : % of runs with irreversible side-effects, privilege violations, data leaks. Must approach zero; any occurrence triggers alert and root-cause analysis.

Human Escalation Rate : % of tasks handed to humans. High rate + high satisfaction = proper boundary setting; high rate + low satisfaction = capability gap.

Policy Violation Rate : % of runs violating predefined policies (unauthorized tool, data access, operation). Core security metric tied to Adversarial Set.

Cost per Successful Task : Average token + tool + infrastructure cost per success. Binds success rate to economics—95% success at $5/task may be worse than 90% at $0.50.

Metrics are interdependent; any sudden shift (e.g., cost doubling while success rate stable) warrants investigation.

Regression Eval: Every Change Triggers It

Any change—model version, provider backend update, temperature, prompt wording, tool schema, skill logic, RAG content/algorithm/embedding/chunking, workflow/router—can cause unpredictable behavioral regression. Automated flow: change commit → auto-run Golden Set → if pass, run Edge Case Set → if pass, run Adversarial Set → if pass, canary → continuous sampling during canary → stable → full release. Requires statistically defined degradation thresholds (e.g., Golden Set pass-rate drop >2% = fail), not gut feel.

Production-to-Eval Flywheel (Agent CI/CD)

Traditional CI/CD: code → build → test → deploy (one-way). Agent CI/CD adds reverse loop: Production → Failure Case Extraction → Human Review & Annotation → Eval Dataset Update → Fix → Regression → Canary → Production. LangChain's Agent Improvement Loop (2026-03-31) validates this: traces from every run feed negative-score review, failure patterns drive code/prompt changes, pre-release offline eval gives before/after comparison, passing evals enter permanent test suite. Microsoft Foundry (Build 2026) productizes this with Trace, Evaluate, Monitor, Optimize capabilities, including "Traces to Dataset" and "Intelligent Trace Sampling". Every production mistake becomes next-cycle training material; without the loop, the same failures recur unseen.

Case Study: E-Commerce Refund Agent Scorecard

Agent handles refund requests: identify issue → query order → select policy → calculate amount → call refund tool → return result. Authorized for ≤¥500; above escalates.

Problem Identification: ≥95% via LLM-as-Judge (Golden 50 + Edge 20)

Order Query Accuracy: ≥98% rule-based (Golden 50)

Policy Selection: ≥90% rule-based (Golden 50 + Edge 15)

Amount Calculation: ≥99% rule-based with ¥0.01 tolerance (Golden 50)

Tool Call Correctness: selection ≥95%, params ≥98%, sequence ≥95% rule-based (all cases)

Permission Boundary: violation rate 0%, escalation ≥99% for >¥500 (Adversarial 30)

Final Resolution: Task Success ≥85%, Escalation ≤15%, Cost ≤¥0.30 (LLM-as-Judge + 10% human spot-check on high-value)

Scorecard evolves: post-launch discovery of "partial refund + coupon stacking" errors adds new Edge Case and permanent regression check.

Eval Platform Minimum Capabilities

Dataset Management: versioned, categorized (Golden/Edge/Adversarial), diffable, bulk import, auto-generation from production logs.

Version Management: model, prompt, tool, dataset versions all traceable; any past eval result reproducible.

Runner: distributed execution, hundreds of concurrent cases, timeout/retry/aggregation, simulates real tools/RAG/permissions.

Judge Engine: configurable judge model/prompt, calibration via human eval, step-rubric support.

Trace Analysis: parse/visualize full trace—reasoning, tool calls, params, returns, latency, tokens. Foundation for trajectory eval and root-cause.

Score Computation: auto-calculate all metrics, slice by scenario/dimension/time.

Regression Automation: CI/CD integration, auto-trigger on changes, auto-compare versions against thresholds, notify team.

Dashboard: visualize agent health—degrading metrics, stale datasets, failed regressions.

Production Sampling: continuous trace sampling, auto-detect failures, push to review queue.

Start with Dataset + Runner + Score; mature by adding Judge, Trace, Regression Automation, Production Sampling.

Seven Common Pitfalls

Only outcome eval, no trajectory eval → misses >50% implicit errors.

Over-reliance on LLM-as-Judge → judge bias, drift, misclassification.

Static eval dataset → new production failure modes never tested.

Regression without thresholds → subjective pass/fail decisions.

Only happy-path cases → agents collapse on real-world anomalies.

No production sampling → unknown unknowns discovered only via user complaints.

Eval metrics disconnected from business KPIs → all-green evals but unhappy stakeholders.

Production Checklist (9 Must-Haves)

Four-layer eval coverage with execution mechanisms.

Three datasets ready: Golden ≥100, Edge ≥50, Adversarial ≥30, all human-reviewed.

Three methods combined with defined applicability, not single-method dependency.

Seven core metrics defined, computable, baselined.

Regression automated for all change types with auto-threshold judgment.

Production-to-eval loop running: sampling, review queue, periodic case review, dataset updates.

Adversarial coverage of injection, escalation, social engineering, exfiltration; security regression blocks release.

Platform basics: dataset versioning, runner, scoring, trace analysis, dashboard.

Judge calibration: periodic human eval, acceptable misjudge rate, full re-eval on judge model change.

Missing any item leaves a blind spot that will become a production incident.

References

LangChain, The Agent Improvement Loop Starts with a Trace, 2026-03-31

LangChain, AI Observability in the Agent Development Lifecycle, 2026-02-10

LangChain, How to Debug & Evaluate AI Agents with Observability

Microsoft Foundry, Build 2026: From observability to ROI for AI agents on any framework

OpenAI, Agent Evals — layered evaluation guide

OpenAI, Enterprise Signals, 2026-08-12 — frontier firms 8.3× token output per employee

Glean, Introducing the Agent Development Lifecycle (ADLC), 2026-05

AgentTrace: A Structured Logging Framework for Agent System Observability, ArXiv 2602.10133, 2026-02