Before Your AI Agent Troubleshoots, Make It Show the Evidence

Opening Example: The Danger of Unevidenced Suggestions

In a procurement scenario, an alert shows a drop in the on-time pass rate for material verification: fewer tasks confirm compliance. The model still returns content and tool interfaces show no widespread errors, yet some tasks repeatedly query contracts without reaching reliable conclusions. An engineer feeds a few log snippets to another agent, which quickly replies: "Model capability may be insufficient; recommend switching models and adding retries." This sounds like a solution but fails to specify which material is missing, when the problem started, or whether retries would actually help.

The previous articles' task records, call chains, and quality logs are now available to verify that suggestion. When asking an agent to help troubleshoot, the first question must be: what evidence supports the claim that the model should be replaced?

Alert Fired: First Confirm What Actually Happened

Start by validating the alert itself. Check whether the statistical definition changed, whether tasks have reached their verification deadline, whether quality checks are delayed, and whether data collection is complete. Until these are ruled out, a dashboard decline cannot be directly attributed to the agent.

Once a real anomaly is confirmed, identify which purchase types and versions are affected, when the issue began, and how they differ from tasks that completed normally at the same time.

Then follow the execution path downstream: did the task get scheduled, which attachments were retrieved, what did the model actually receive, how far did the tool progress, and are there already-completed actions in the business system. Query the corresponding systems only when you have specific clues, avoiding full log scans every time.

If critical business is already impacted, execute the mitigation plan first — the responsible person decides which task types to pause or what controlled degradation to apply — without waiting for full root-cause confirmation. However, effective mitigation and confirmed root cause remain two separate things.

Google's Site Reliability Engineering describes troubleshooting as a process of proposing hypotheses, checking evidence, and using tests to narrow scope. This method applies equally to model-assisted troubleshooting; a complete-sounding explanation must not replace verification. Source: Effective Troubleshooting.

Fixed Queries to Programs, Model Organizes Clues

Task status, call chains, release and config versions, and exception distributions typically have fixed query entry points. Implement these as deterministic programs, then hand the results to the model for organization and analysis of what to investigate next.

Wang Zefeng, in "AI Coding Speedup: How to Close the Stability 'Last Mile'? Kuaishou Conan AI Practice" (InfoQ talk summary), describes a division of labor: codify clear processes, use single model requests for simple analysis, and employ agents for complex exploration. This approach — choosing execution mode by problem type — is well aligned.

For the procurement anomaly, the minimum evidence package given to the model must include:

Affected and normal control task identifiers, stages, and wait times.

Corresponding call chains, error records, retrieval results, and business object versions.

Recent changes to application, model, prompts, index, and retrieval configuration.

Checks already performed, their results, and data still unobtainable.

Preserve source, query conditions, time range, and collection timestamp for later cross-checking. Crucially, do not lose original meaning during summarization: "attachment not found" and "permission denied, could not query" imply completely different investigation directions.

If the material is sufficient, one analysis round should conclude; if gaps remain, allow the agent to supplement queries within a defined scope. Whether a long reasoning loop was used is not the criterion for judging solution quality.

List Causes as Hypotheses, Confirm with Evidence

Suppose investigation finds anomalies concentrated after a retrieval configuration update (K-12), with some tasks reading old attachments. "Retrieval configuration issue" is a reasonable hypothesis, but not yet a confirmed conclusion.

The troubleshooting process can be recorded in a structured template. The following example uses placeholder timestamps, IDs, and observations; verification results are still pending.

Hypothesis Verification Record Template

Confirmed Facts : 14:00 released retrieval config K-12; sampled anomalous tasks returned old attachments and repeatedly re-queried. Linked to change records and execution records.

Candidate Cause : K-12 may have caused some requests to still access the old index, so effective attachments were not retrieved.

Supporting Clues : Anomaly time close to config change; query records of these samples point to old index.

Unexplained Gaps : Whether all anomalous tasks are affected by the same issue; whether similar failures occurred under the old config.

Next Checks & Counter-evidence : Verify actual routing at failure time, then compare configs in an isolated environment; if online records prove routing was correct, revise the old-index hypothesis and continue investigating where attachments were filtered or missed.

Responsibility & Remediation : Retrieval service owner verifies routing; config switch decision by authorized personnel per runbook, recording preconditions and impact scope.

Verification Result : Pending: routing verification, comparison test, remediation result, observation window. Remains a candidate cause until results are filled.

With this record, the next investigator can continue and knows what evidence would require a direction change.

An unchanged model version only lowers the probability that a model upgrade introduced the problem; it does not prove the generation step is flawless. Similarly, a config change coinciding with the alert may simply coincide with tasks becoming more complex or dependencies slowing down.

If the isolated environment fails to reproduce, first verify whether configuration, data, and permissions match the production state at the time of failure. Changed conditions naturally prevent reproducing the old issue. Record which tasks are explained and which remain unclear separately; do not rush to assign a single common root cause to all anomalies.

Read-Only Queries Can Still Add Chaos

The troubleshooting agent defaults to read-only. Yet read-only operations have cost: repeatedly scanning large log volumes or executing unbounded database queries can exacerbate the failure.

Query tools must therefore enforce limits: tenant, task scope, time window, result count, timeout, total query count, and cost budget. When queries are insufficient, the agent may propose expanding scope, but the responsible person must approve; the model cannot self-grant permissions.

Logs, tickets, and tool returns are analysis material only. Even if a log contains "run this command to recover," that does not become an executable instruction. Query failures, permission denials, and empty results must be distinguished so the model does not conflate "not visible" with "does not exist."

When sources conflict, supplement time, version, and source queries within the established permissions and budget. If key conflicts persist, necessary evidence is unobtainable, repeated queries yield no new findings, or the budget is exhausted, stop automated diagnosis and hand the current conclusions, gaps, and next-step recommendations to the specific responsible person.

Each troubleshooting execution receives a unique ID recording what was queried, which version was used, the basis for each step, which suggestions were modified or rejected, and links to affected tasks. Otherwise, business agent and troubleshooting agent calls become interleaved, making later traceability harder.

Query Is Query; Config Changes Require Authorization

When the model proposes "switch back to old index," "restart service," or "replay tasks," each requires further checks of current state, impact scope, and operation permissions.

Any automated remediation must be backed by a reviewed runbook — a runbook with explicit steps and boundaries. It must specify applicable failure types, allowed objects and actions, pre-execution checks, stop conditions, recovery procedures, and the responsible person. A documented runbook is not itself an execution permit; the corresponding authorization must be obtained.

For example, reverting to the old retrieval configuration first requires verifying that the target index still contains applicable materials and that permission rules are correct. A configuration that worked in the past may not suit today's data.

For tasks with unknown submission results, repeated execution cannot be a universal fix. The original action and external state must be verified first; any re-submission still requires checking current business preconditions and authorization.

Before every execution, the program validates current conditions and valid authorization — not just the prompt. If state changes invalidate preconditions, the operation exceeds authorized scope, or remediation shows no improvement within the agreed observation window, automated progression stops and escalation to a human occurs. The agent must not keep trying larger-scope operations.

After a fix, verify that tasks actually use the correct attachments. If the interface is faster but answers still reference old materials, the fix has not solved the business problem. Once new tasks run normally, previously affected tasks must still be processed.

How to Prove the Agent Actually Helped

Before production deployment, test in an isolated environment with two known issues: a tool dependency timeout and a retrieval configuration pointing to an old index.

Check whether the agent finds the corresponding evidence and distinguishes the two causes, rather than suggesting retries for any anomaly. Remove key logs or receipts and observe whether it explicitly states it cannot confirm. Add evidence contradicting the initial hypothesis and verify whether it corrects its conclusion.

Also test whether out-of-bounds suggestions are blocked, whether queries respect the budget, and whether remediation stops when ineffective. Do not merely test whether the diagnostic text sounds reasonable.

For effectiveness evaluation, do not rely solely on suggestion adoption rate. An engineer's willingness to try a suggestion and the later proof that the root cause was correctly identified are different metrics. More valuable to record: conclusion verification accuracy, count of misleading suggestions and human corrections, time to first valid clue, and time to business recovery.

These times need clearly defined start and end points and should be compared across incidents of similar difficulty. Unconfirmed cases must be listed separately and not counted as correct judgments. Human review and verification time, query costs, and model costs must be included in the investment calculation.

If the team lacks complete task records and query entry points, fixing those is more valuable than deploying a troubleshooting robot. Faults that fixed checks can already locate do not need forced model involvement.

The goal is to reduce the time spent repeatedly reading logs, checking versions, and piecing together the scene. An agent that assembles clues, explains its reasoning, and hands over gaps when unclear is already worthwhile. Deciding which system to touch and what risk to bear remains a human responsibility.