Agent Calls Succeed But Answers Fail: Tracing Observability to Quality Evaluation

An Agent processed a procurement request for a 50,000 RMB prepayment against a contract whose current effective attachment capped prepayments at 30,000 RMB. The Agent cited the old attachment (v1, 50,000 RMB limit) and concluded the request was compliant. No tool errors, timeouts, or retrieval failures appeared in the trace — the call chain looked clean, yet the conclusion was unusable.

Locate the Faulty Stage Before Scoring

Using the observability platform, the author examines four points at the time of the task: the applicable materials, retrieval results, actual model input, and final citations. Four distinct failure modes are possible:

The effective attachment (v2) was never indexed, so retrieval missed it.

The attachment was indexed and accessible but excluded by a wrong query filter or permission rule.

Retrieval returned the attachment, but context assembly or compression dropped the key clause.

The key clause reached the model input, yet the model still used the old amount.

Each mode requires a different fix. Recording only "answer wrong" leads to prompt-tweaking guesswork. Verification must return to the task-time objects, versions, effective dates, and material availability. A document provided today cannot count as yesterday's retrieval miss, but a document that was effective and should have been ingested per agreement remains an ingestion issue. When records are insufficient, mark as unconfirmed — do not let the model hallucinate a cause. Traces may not store full materials; follow references to reconstruct inputs and results, and list any missing parts as pending investigation without premature conclusions.

Turn One Feedback Item into a Reproducible Test Sample

"User says answer is wrong" is a clue, not a ground truth. First confirm the business rule, applicable materials, and the actual deviation, then decide whether to add it to the evaluation set. The article provides a concrete sample structure (illustrative IDs and content):

Source : original task, execution, trace references, and verified user feedback.

Input & applicable basis : 50,000 RMB prepayment; contract C-17; at that time, effective and accessible attachment v2 with 30,000 RMB limit.

Test environment : provide v1 and v2 materials with their effective relationships; v2 is searchable; submission tool uses a mocked implementation that records calls.

Observed deviation : Agent cited old attachment v1's 50,000 RMB limit and judged compliant.

Expected behavior : find and cite v2, flag over-limit, pause submission, prompt adjustment or escalate per business rule; merely answering "pending verification" does not pass.

Hard checks : correct object and attachment version, correct amount comparison, must not submit when conditions unmet.

Semantic review : did it explain the over-limit reason? did it state uncertain content as confirmed fact?

Regression record : versions of input materials, model, prompt, retrieval config, evaluation rules, plus each run's result and review notes.

Once expected behavior is explicit, the Agent need not reproduce a canonical answer verbatim — alternative phrasing or a different valid query path is acceptable if basis and action are correct. Before adding to the suite, confirm usage rights, deduplicate, and anonymize. A subtle point: contract IDs and supplier names can be replaced, but object relationships and amount constraints must stay consistent, otherwise anonymization changes the problem.

Use Rules for Deterministic Checks, Model Assistance for Complex Judgment

Referencing Anthropic's "Demystifying evals for AI agents", the article distinguishes three judging modes: programmatic, model-assisted, and human. Contract IDs, attachment versions, currency, amounts, and submission logs are deterministic — programmatic checks work here, provided a trusted reference exists. Comparing two model-guessed values against each other cannot reveal if both are wrong.

Programmatic checks confirm "50,000 > 30,000" but may not independently interpret complex clause exceptions. For those, an evaluation model can assist, required to cite evidence, state reasoning, and note missing information. However, evaluator models also err. The paper "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" documents position and verbosity biases; longer, expert-style answers do not guarantee accuracy. The evaluator itself must be tested on business-verified samples to measure false passes and false fails. Disagreements between human and model reviewers go back to materials and rules; unresolved samples are held as pending.

Swapping the scoring model does not equal independent verification. Judgment criteria must come from confirmed materials and acceptance requirements; the answer under test is only the content to be checked, not a signal to steer the evaluator. Evaluation rules, prompts, and model versions must be recorded, otherwise score changes may reflect shifted judging standards rather than Agent improvement.

Don't Let a Single Aggregate Score Mask Critical Errors

Four dimensions must be scored separately: whether required materials were retrieved, whether conclusions have basis, whether the task was completed, and whether execution stayed within permissions. Fluent, well-cited output that reads 50,000 as 5,000 is still wrong. A correct conclusion does not cancel out an unauthorized read of another supplier's contract.

Before trusting scores for version comparisons, sanity-check the evaluator on simple counter-examples:

Change prepayment from 5,000 to 50,000 (other text unchanged) → judgment should flip from under-limit to over-limit; character overlap cannot decide correctness.

Keep real citations but swap in another contract's attachment → real material ≠ applicable to current object.

Text says "pause submission" but tool already submitted → explanation inconsistent with action.

Amounts, basis, and actions all correct, only wording changed → should still pass, avoiding overfitting to fixed phrasing.

Also retain samples with incomplete materials. Correct behavior may be to re-query or ask for help; do not force a definitive conclusion for the sake of "completion rate". Conversely, when materials are sufficient and rules satisfied, blanket refusal is not higher quality. Verdicts: pass, fail, or indeterminate. If materials are missing, check whether the Agent correctly re-queried or sought help; if materials are complete but answer is wrong, it's a fail. If evaluation evidence is missing or the evaluator faults, mark as temporarily indeterminate with reason — do not blame the Agent nor default to pass.

Regression Suite Must Include More Than Past Failures

Collecting only known failures (e.g., missed attachments) risks over-correction: the Agent may start rejecting every case with an attachment, blocking originally valid procurements. The regression set therefore needs:

Failure cases

Normal success cases

Reasonable rejections

Boundary cases

Group by material complexity, risk, and anomaly type. Deliberately hard cases probe capability limits; their pass rate is not production success rate. Debugging samples and final acceptance samples must be separated. Near-duplicate variants of the same contract belong in the same group to avoid tuning on one and claiming independent test pass on another. Expected answers must never leak into the Agent's input. Once an acceptance sample is used for debugging, retire it and add fresh samples.

Model execution varies; important cases should be re-run a pre-agreed number of times, preserving every result. Compare quality, human review outcome, and cost — but do not let low cost offset critical business errors.

Before Rerun, Define Exactly What You Are Verifying

If only checking model comprehension of an attachment, fix retrieval results and compare models or prompts. This validates "given these materials, how does it answer" — it does not prove retrieval is fixed.

To verify the attachment miss, include the retrieval stage: prepare the corresponding material versions, index, and permission conditions. After changing retrieval strategy, check both that the correct attachment is found and that unauthorized materials are not returned.

For steps involving submission approvals or notifications, isolate production writes using a test system or controlled mock tools; log every call, parameters, and result. A blocked violation call must be reported separately from "Agent did not attempt violation" to pinpoint the source.

Before each rerun, reset session, business state, and mock receipts to the agreed starting point, preventing leftovers from a previous run from completing the next task.

Rerun results are new outcomes under current conditions. Even with identical inputs, external services, model versions, and execution paths may differ — do not record them as a full reproduction of the original scene. Link evaluation results to the original task and sample version, store as a new test run. Today's pass does not erase yesterday's failure; production submissions still require original authorization checks.

Starting small is fine. Begin with the old-attachment misuse: document the basis and expected behavior, add a few originally correct cases. Run them together after each change — at minimum you'll know whether the old issue is fixed and whether new issues appeared. These records, beyond regression, can also be fed to an Agent to help engineers debug. The next article covers how to do that.