Agent Observability: From Traces to Task Failure Attribution — Production Design with Go, Java, OpenTelemetry & Kubernetes

Introduction

The article addresses the challenge of observing long-running agent tasks that span multiple processes, traces, approval cycles, and uncertain external calls. It argues that traditional HTTP APM is insufficient because an API response only indicates task acceptance, not completion. The core constraint: task fact storage proves state, while observability explains state; they share correlation keys but cannot replace each other.

Core Concepts: Task, Step, Attempt, Invocation, Effect

The author defines a precise hierarchy: Task (user-submitted logical work, immutable task_id), Step (persisted logical action, task_id + step_id), Execution Attempt (worker's execution interval, execution_id), Step Attempt (single try of a step, step_attempt_id), Invocation (actual LLM/MCP network request, invocation_id), Effect (logical side effect, effect_id / idempotency_key), and Trace (set of spans, trace_id allowed to change across attempts). This separation avoids conflating retries, costs, and idempotency.

Five Signal Types: Traces, Metrics, Logs, Events, Audit

A table maps each signal to its primary question, examples, and reliability boundary. Traces show what was called and where time was spent (may be sampled/lost). Metrics show aggregate trends (cannot prove single-task facts). Logs capture detailed operation context (stdout ≠ reliable audit). Events record state changes (should be generated from transactional facts). Audit records who attempted what under which authorization (independent permissions, integrity protection). The article warns against inferring single-task call order from aggregate metrics.

Architecture: Business Evidence Chain vs Telemetry Pipeline

Two distinct data paths: (1) Traces, logs, metrics may degrade within budget and must not block business. (2) State transitions, effect records, and critical audit must meet business reliability; high-risk tools should refuse execution if authorization/effect cannot be persisted. An Outbox pattern ensures database state and events commit together, but does not guarantee external tool exactly-once execution.

Context Propagation Across API, Queue, Worker, MCP

API writes Task and Outbox in one transaction, returns 202. Worker creates a new root trace with WithNewRoot() and uses Links to connect to the triggering message and previous execution (Go code provided). HTTP uses W3C headers via otelhttp; MCP 2026-07-28 spec reserves traceparent, tracestate, baggage in _meta — the adapter must merge trace context with SDK-generated protocol metadata, not overwrite. Recovery does not replay the whole task; it links new execution to prior context and verifies effects via idempotency keys.

Crash Recovery and Execution Takeover

Crash processes cannot reliably write final spans (SIGKILL/OOMKilled). Therefore: persist execution start, ownership transfer, state transitions; external controller collects Pod/Container/Node evidence; recovery records "observed lease expired and took over" not "old worker OOM". Correlation requires cluster, namespace, pod UID, container, restart count, time window. Status events express uncertainty: termination_cause: unknown until container evidence (e.g., reason=OOMKilled) is attached later via a separate event, preserving original record.

Lease, Fencing, Checkpoint, Outbox Observability

Kubernetes Lease does not provide business fencing; the article uses PostgreSQL row-level fencing_token per task. Minimal fact tables ( agent_task, agent_execution, agent_event_outbox) with CAS updates (

WHERE current_execution_id = $3 AND fencing_token = $5 AND checkpoint_version = $6

). Checkpoint commit must verify current token equality (not <=), and Go code must check RowsAffected() == 1. Each mechanism observes different failure facets: lease (ownership observation), fencing (downstream rejection), checkpoint (commit latency/version/failure), outbox (pending age, publish retries), effect ledger (unknown effects, deduplication, confirmation latency).

Go Instrumentation: Initialization, Error Handling, Structured Logging

SDK initialization locks API/SDK/exporter versions in go.mod. Shutdown uses a separate timeout context. RecordError does not replace SetStatus; errors are categorized (e.g., llm_rate_limited, tool_auth_denied, effect_unknown) to drive retry strategy. Panic values are sanitized (only type logged) to avoid leaking credentials. A wrapper ObserveOperation records error category, sets span status, logs structured fields ( task_id, execution_id, trace_id, error_category), then re-panics for upper-layer handling.

LLM Token, Latency, Cost: Three Distinct Measurements

Latency: non-streaming records total; streaming records time-to-first-token (TTFT), generation time, total. TTFT missing for pure tool calls — not zero. Token usage: prefer provider-reported usage; client-estimated marked estimated; missing due to cancel/timeout marked missing. Cost is a ledger calculation: C_estimated = Σ U_k × P_k(model, region, price_version) with non-overlapping billing units. Ledger records invocation_id, provider request ID, model resolution version, usage source, categorized units, currency, price version, estimated cost, reconciliation status. Custom Prometheus metrics ( agent_llm_request_duration_seconds, agent_llm_time_to_first_token_seconds, agent_llm_tokens_total, agent_llm_usage_missing_total, agent_llm_estimated_cost_usd_total) with bounded labels (workflow, model, outcome, token_type, usage_source, reason).

MCP Authorization, Approval, Audit, and Effect Closure

Tool success ≠ authorization success. Chain:

task_id → step_id → invocation_id → authorization_decision_id → approval_id/version → effect_id → external_receipt_ref

. Audit distinguishes attempted, accepted, confirmed, rejected, unknown. Approval binds normalized tool parameters, target resource, operation version; recovery must re-check authorization if any change. For non-idempotent tools, the dangerous window is "remote success, local commit before crash"; solutions: idempotency key reuse, result query, or effect_unknown with manual reconciliation. Fencing only protects resources that validate tokens.

Collector and Kubernetes: Making the Observability Pipeline Observable

Collector Contrib config template (OTLP/gRPC 4317, Tempo trace export, Prometheus metrics on 8889, memory limiter, batch processor, file storage persistent queue). Must pin collector image, validate config with that image, mount persistent volume for /var/lib/otelcol/storage (not emptyDir). Worker identity injected via Downward API ( POD_UID, POD_NAME, POD_NAMESPACE) into OTEL_RESOURCE_ATTRIBUTES. Graceful termination: stop claiming, revoke readiness, drain/persist in-flight, then shutdown OTel with independent timeout.

PromQL and Grafana Dashboards: From Aggregate to Single Task

Queries assume custom metrics contract. User task failure rate excludes cancellations, uses agent_task_terminal_total from deduplicated terminal events. MCP tool call P99 uses histogram_quantile with le label preserved. Model usage and cost from increase() on counters (trend estimates, not bills). Lease/checkpoint/outbox metrics: takeover reasons, checkpoint commit failures, oldest ready outbox age. Dashboard drill-down: user results → workflow/time; execution health → worker/version/logs; LLM → model → trace/ledger; MCP/audit → audit decision → effect; infra/release → pod UID/release; observability quality → collector instance. Single-task evidence page queries Task Store for all execution trace IDs; Grafana jumps must enforce authorization.

Argo CD Release Correlation: Time Proximity Is Only a Clue

Three evidence layers: Argo CD sync (desired revision, app, env, result), Deployment/ReplicaSet template changes and rollout status, actual pod image digest, app version, config/policy digest. Argo CD Notifications trigger on sync success + healthy, once per revision, sends webhook with app UID, revision, release ID. Correlation ladder: failure rate overlaps release annotation (time correlation) → failures concentrate on new version pods (version correlation) → new version changes lease renewal path, logs show renewal anomalies (mechanism explanation) → same-input reproduction, fix/rollback makes symptom disappear (stronger causality). Recovery time splits into remaining lease wait, scheduling, cold start, state read, effect verification — not sum of P99s.

Task Failure Attribution: Output Conclusions and Evidence Gaps

On-call workflow: (1) check task state/version/last progress; (2) list all executions, step attempts, invocations; (3) find first deviation (auth denial, dependency error, ownership change, unknown effect); (4) cross-check auth, approval, effect ledger; (5) attach Kubernetes termination evidence by container instance/time window; (6) verify actual execution version/config/release; (7) record unproven parts and next verification. Attribution template (YAML) includes candidate_cause: unknown, confidence: insufficient, facts with source/ref, trace/audit/effect/k8s/release refs, missing evidence, alternative explanations. LLM may assist but must cite accessible evidence IDs and allow unknown.

Fault Injection Experiments: Design and Verification

Eight experiments (E1–E8) in isolated namespace with pinned versions: E1 forced termination after execution context commit before checkpoint (new execution, takeover event, new trace link); E2 tool effect committed but response lost ( effect_unknown, original effect ID, effect query); E3 old owner writes after lease loss (new token, old condition updates 0 rows); E4 collector disconnect during execution (export queue/discard, business events intact); E5 approval change after grant (re-check auth, reject audit); E6 stream interruption after first token (error, missing usage, cost pending); E7 rolling update during run (release/pod/execution correlation, stop claiming, no old owner commit); E8 malicious context input (parse result, local auth/limits). Each has injection point, required observations, pass criteria. SQL verification queries for execution, task, outbox, effect ledger. Experiment record template captures version manifest, IDs, barrier timestamps, raw evidence, expected/actual, judgment, gaps.

Sampling, Privacy, and Cost: Complete Attribution Without Infinite Collection

Per-task consistent sampling is an application design: decide at task creation via hash of task_id with secret, propagate telemetry_sampled in persisted context, custom sampler reads at root span creation. Head sampling discards spans tail sampling cannot recover; even 100% sampling loses in-flight spans on crash. Therefore state events, critical audit, usage/cost ledger retain per their reliability requirements. Volume budgeting: task count × executions × spans × span size × retention. Compress low-value repeats, bound tool output length, use bounded labels/versioned fields. Hashing ≠ anonymization (short enums enumerable); use keyed HMAC with rotation. Baggage whitelist with size/count limits; cross-domain strips tenant info. Trace, logs, audit, raw content have separate access control/retention.

From Minimal Loop to Production Acceptance

Rollout sequence: (1) unify identities and error enums; (2) persist execution context before side effects, wire propagation; (3) implement takeover events, checkpoint CAS, effect verification; (4) add independent metrics, structured logs, audit, verify facts survive trace loss; (5) add Kubernetes container evidence, release IDs, Argo CD annotations; (6) run E1–E8, verify sampling, backend failure, data permission boundaries. Acceptance criteria: random task ID yields all execution intervals, step attempts, external effects; missing old trace explained (sampling/export/undetermined); old owner revival protected by state and external write mechanisms; approval expiry, tool business failure, usage missing have distinct measurements; cluster release locates affected container instances/versions; observability backend down — task facts and critical audit still saved per contract.

Appendix A: Java Reference Implementation

Same task model, different SDK lifecycle. Java snippet creates new root span with setNoParent(), adds links from previous/trigger carriers via W3C propagator, sets attributes ( aegis.task.id, aegis.execution.id, aegis.execution.attempt). Scope limited to current execution; thread pools must explicitly propagate Context. Shutdown order: stop claiming → revoke readiness → cancel/persist in-flight → shutdown thread pools → flush/shutdown OTel with separate timeout. CompletableFuture /Reactor/coroutine context propagation must be integration-tested; explicit task_id, execution_id structured fields remain necessary fallback.

Appendix B: Deliverables, Acceptance Evidence, and Definition of Done

Table maps deliverables (task fact schema/migrations, Go/Java instrumentation libs, MCP adapter, effect ledger, collector/backend, dashboard/permissions, fault experiments) to minimum completion standards (automated tests for concurrent claim/old owner write/unknown effect; locked deps, compile, propagation tests; SDK negotiation metadata preserved, failure classification; idempotent query/duplicate delivery/unknown path regression tests; image config validation, persistent volume, queue/discard/disk alerts exercised; task drill-down, tenant isolation, trace/log/audit jump auth tests; E1–E8 versioned raw evidence archived) vs insufficient signs (few tables created, local span visible, HTTP 200 only, effect ID only in logs, pod running, panel variable filters task ID, drill oral pass). E1, E2, E3, E5, E7 recommended as pre-release gates.