Data Bricklaying Diary

Oct 17, 2026 · Artificial Intelligence

Agent Calls Succeed But Answers Fail: Tracing Observability to Quality Evaluation

The article explains how successful Agent execution traces can still produce wrong answers, using a procurement case where an Agent cited an outdated contract attachment. It details a systematic approach: trace errors through observability across indexing, retrieval, context assembly, and generation; convert feedback into reproducible test samples with versioned inputs and expected behaviors; evaluate using rule-based checks, model-assisted judging, and human review; separate critical error categories instead of relying on a single score; maintain regression suites with both failure and success cases; and rerun tests in isolated environments while recording versions to distinguish new results from original incidents.