Agent Observability: Correlating Model, Tools, and Business Receipts to Verify Task Completion

When building microservices, the author used OpenTelemetry for data collection, Jaeger for trace inspection, Prometheus and Grafana for metrics, and Graylog for centralized logs. When services slowed or errored, these signals enabled root-cause analysis. Now developing Agents, the author faces new questions: why does the Agent keep calling tools, where is the money going, and did the assigned task actually complete? This gap is also discussed in the author's previous article "功能完成不等于生产就绪：AI 编程为什么更需要工程化" (Feature Completion Does Not Equal Production Readiness: Why AI Programming Needs More Engineering). The goal of Agent observability is to determine what the Agent has done, where it is stuck, and what evidence supports its results.

Interfaces Are Healthy, Yet the Task Remains Unfinished

Consider a procurement-approval task: the Agent must verify budget, supplier, and contract, submit for approval, and notify the requester. The UI shows "processing" while API success rates and server dashboards look normal. At least four distinct states can produce this same symptom:

Materials verified, waiting for user confirmation.

Approval accepted, waiting for approver action.

Tool returns irrelevant materials, causing the Agent to re-query repeatedly.

Submission timed out; the external system may have accepted it but no receipt was captured locally.

All appear as "not done" yet require different responses: wait, investigate, or stop wasteful retries. Traditional systems also have async waits and state inconsistency, but Agents add non-deterministic execution paths that change with retrieval content, model output, and tool results. A successful request may only be the start of troubleshooting.

What Existing Tools Still Reveal — and What They Miss

Logs record events, metrics show trends, and traces (call chains) track component hops and latencies. These remain useful for Agent debugging. However, default HTTP and database instrumentation cannot answer whether a query found the applicable contract, or whether a tool's "success" means request accepted or approval completed.

For the procurement task, after examining the trace we still need answers to five questions, each requiring specific evidence:

Where did execution stop and why? — Current phase, wait reason, entry timestamp, responsible party.

Where was time spent? — Queueing, model generation, retrieval, tool execution, human wait durations.

Why did cost increase? — Per-call model usage, retries, sub-task and tool overhead.

What is the basis for the answer? — Retrieval citations, business object versions, rule checks and review results.

Did the action actually occur? — Action identifiers, business-system receipts, subsequent query results.

The article "Enterprise AI Application Observability: Metrics System and Instrumentation" proposes monitoring latency, cost, and quality together. Quality, unlike latency, cannot be measured directly; it requires defining and validating judgment methods. Complex contracts may need supplementary checks, submissions may require confirmation. High call counts or human involvement do not necessarily indicate inefficiency. When comparing optimizations, failure and retry costs must be included alongside delivery results and manual rework. Low token usage does not guarantee better task outcomes.

Connecting Runtime Signals with Business Facts

The author proposes a design where the existing platform continues to ingest runtime signals (logs, metrics, traces), while the Agent runtime emits model, retrieval, and tool events. The control plane — the subsystem managing task state, permissions, and execution boundaries — contributes authorization, wait, resume, and result-verification events. When viewing a task, these are correlated by a stable task identifier.

Data may be stored separately, but must be discoverable from the task. The task record should also link the model version, prompt, retrieval index, and tool version active at execution time; otherwise it is difficult to determine whether a regression originated from a specific change. A single task may span multiple conversation turns, async executions, and even service restarts, mapping to multiple traces. OpenTelemetry's trace specification provides mechanisms to correlate execution fragments, but technical correlation alone does not define task completion.

Recording reliability is critical. Traces can be sampled, but user confirmations, submission intents, and external receipts must be persisted per business requirements. If a trace segment is missing in Jaeger, we cannot distinguish between data loss and an action that never happened. A model reply of "submitted" cannot replace the missing receipt.

Seeing the Process Is Not Enough — Results Must Be Validated

Suppose the trace shows the Agent quickly completed contract retrieval and answer generation at low cost. Yet it read an outdated attachment and missed the latest prepayment constraint. This error does not appear in latency or cost metrics. The task view must surface retrieval citations, object versions, and check results: explicit conditions (amount, object, status) verified programmatically; complex contract interpretations assisted by evaluation models and human review.

These records reconstruct visible inputs, calls, and outputs, but cannot expose the model's full internal reasoning. Visibility into external models depends on what the provider surfaces.

Qian Shijun's "Giving Agents a CT: Observability and Quality Assurance for Large-Scale Agents" discusses feeding production traces back into evaluation. For this case, the value lies in turning the outdated-attachment mistake into a regression sample: when adjusting retrieval or swapping models, verify the same issue does not reappear. Samples must be verified, de-identified, and labeled; representative successes should also be retained to avoid over-optimizing only for failures. Replay involving write operations (e.g., submitting approvals) must use controlled environments or fixed receipts, not create real approvals for debugging.

Start by Clarifying One Task

Read-only assistants can begin with source attribution, permissions, latency, and feedback. Procurement approvals involve writes and long waits, so action verification and resume records are essential. Start with a single task, defining completion conditions, query entry points, and exception validation.

Since the task includes result notification, tracking must extend to notification delivery. Normal wait durations, escalation owners, and timeout handling should be pre-defined. Alerts sent to owners must include the task ID and query entry point so they know where to start investigating.

The query entry can be simple: given a task ID, retrieve model and tool calls, current phase, external actions, and supporting evidence. Sensitive materials should be referenced via controlled pointers rather than duplicated across every log store.

Once integrated, test in a staging environment: force tool timeouts, pause and resume tasks, return stale retrieval results — verify the records reveal the cause. Then break the collection pipeline and confirm the UI signals missing data instead of showing "all normal."

These responsibilities should be assigned at design time: architects define observability scope and evidence requirements; platform teams provide common ingestion capabilities; developers add task-specific instrumentation; business owners confirm result standards. Small teams may have one person wear multiple hats, but each role must have an owner.

With Evidence in Place, Let Agents Assist Troubleshooting

With this data, Agents can summarize anomalies, query records, and propose hypotheses. Kuaishou's "After AI Coding Acceleration, How to Close the Stability 'Last Mile'?" offers a reference: deterministic steps executed by code, analytical judgments assisted by models.

However, using Agents for troubleshooting and making Agents themselves observable are distinct efforts. Without sources, versions, and actual receipts, even a powerful analysis model can only speculate from incomplete information. If diagnostic suggestions are to become production actions, they require corresponding authorization and execution checks.

When a user says "this task wasn't handled well," the author wants engineers to start investigating immediately with just the task ID. The next article will begin with the most common "processing" state, examining how to make progress and wait reasons explicit.

References

Qian Shijun: "Giving Agents a CT: Observability and Quality Assurance for Large-Scale Agents." https://www.infoq.cn/article/QqDSrCEGWitk7kkJg7Kj

Chuan Guo Sheng Fang Fen Fang: "Enterprise AI Application Observability: Metrics System and Instrumentation." https://xie.infoq.cn/article/0c0ada9c1ae9880c8caca1214

Wang Zefeng: "After AI Coding Acceleration, How to Close the Stability 'Last Mile'? Kuaishou Conan AI Practice." https://www.infoq.cn/article/rlXYaPUjVR0fffbQK3Iz

OpenTelemetry: Traces. https://opentelemetry.io/docs/concepts/signals/traces/