One Task, Multiple Traces: Connecting Agent Sessions, Tools & Async Execution

Task IDs and Trace IDs Serve Different Purposes

A purchase task may span two days, involving user confirmation, process restarts, and approval callbacks. It does not map to a single trace. The article defines distinct identifiers: app.task.id – the whole purchase task and its execution records; identity persists across pauses and resumes. app.session.id – the interactive session at that moment; a session may involve multiple tasks, and background execution may have no current session. app.execution.id – one scheduled execution round; a new record is created on resume, preserving its link to the previous round. app.action.id – a business action that needs outcome tracking (e.g., submit approval); retries and queries link back to it, while new business intent gets a new action ID. app.attempt.id – a concrete attempt of an operation; separates first call from retries for result verification. trace_id / span_id – the call chain and operation fragments, managed by the tracing system, not used as long-term business keys.

One execution round can contain multiple model or tool calls, each retry being a separate attempt. A timeout on approval submission must link the verification execution to the original action and note it is a query, not a resubmission. These business links are maintained by the task service and executor; the tracing component only generates trace identifiers.

A Task Can Have Several Self-Contained Execution Segments

For long-wait tasks, the author prefers letting each actual execution segment finish promptly and stitching them together via the task record. The mermaid flowchart illustrates a purchase task (T-42) with three traces:

flowchart TB
  T[Purchase Task T-42]
  T -. associated .-> A[Trace A: verify materials, send confirmation request]
  T -. associated .-> B[Trace B: submit approval after confirmation]
  T -. associated .-> C[Trace C: query original submission result]
  A --> W[wait for user confirmation, current execution ends]
  W --> B
  B --> R[submission timeout, record original action and schedule verification]
  R --> C
  C --> E[found approval instance, link to original submission action]

Human waiting overnight does not require keeping the root span open; wait time is derived from task state, while model/tool latency uses actual call durations. OpenTelemetry Span Links allow a new trace to reference the triggering old execution fragment, expressing causal links beyond parent-child. Direct calls use real parent-child spans; cross-segment or multi-source merges use Links. Short async ops may stay in the same trace; splitting should follow the instrumentation scheme, not mechanically by model turns or queue hops.

Context Propagation Is Most Fragile at Handovers

Three handover points break correlation most often:

Enqueue/dequeue : scheduling messages must carry trusted task/execution links and trace context. Consumers create their own operation records, log message source, and treat duplicate deliveries as separate attempts without overwriting old records.

Pause/resume : durable records must store business links needed for recovery and any usable trigger span context. New execution continues tracing per actual relationships or creates a Link. After the old process exits, memory-only context is lost; if old span info is missing, preserve the task link and annotate the trace gap.

Concurrency and merge : parallel branches (budget, vendor, contract) use independent execution contexts. If split into subtasks, retain parent task and branch IDs. The merge records which results were received; Links can associate multiple sources — never guess ownership by return order.

Propagated identifiers are for correlation only, not authorization. At tenant boundaries and external callbacks, the system must still verify identity, ownership, and access scope.

Instrumentation Must Sit Where Work Actually Happens

Instrumenting only the model gateway shows model calls but hides why the task waited, who resumed execution, or why a result was rejected. Scheduling, pause, resume, and state changes belong in the runtime and task manager. Model requests, retrievals, and tool calls are instrumented at their true execution points.

Operation records should include name, start/end times, result, necessary input/output references, and the application release version. Model, prompt, index, and tool versions are added per operation. Minimal model call data: model ID, prompt template version, latency, and service-reported usage; missing usage is marked absent (not zero) to avoid undercounting cost.

Retrieval records must answer "what was found at that time": controlled query reference, filter conditions, index version, and the returned document or chunk versions. For tool calls, separate the call result (HTTP status, latency) from the business result (e.g., "materials incomplete"). Preserve parameter summaries or controlled references, and link to the action, receipt, and subsequent verification.

Expensive steps like context compression or memory loading get their own spans. Model switches or retry decisions are recorded as events with the triggering rule. A runtime switching models due to a timeout logs the rule; a model claiming "I changed method for accuracy" is only explanatory text, not a verified cause.

Prefer OpenTelemetry semantic conventions for common fields; GenAI conventions exist but require version and SDK support checks. Custom business fields are managed separately. Do not collect full prompts, contracts, or tool responses by default; keep metadata and controlled references for debugging, fetch content on demand with permissions, and avoid leaking sensitive data via trace context.

Even When a Trace Segment Is Missing, Know Where the Gap Is

External approval systems may not share the same tracing stack. Record the local request, the remote request ID or approval instance, and link subsequent queries. Explicitly stop at that boundary. Sampling, export failures, or retention limits can also make a trace segment unavailable. Links only store associations; they cannot guarantee the linked data exists or that UIs will auto-render a complete task graph.

Therefore, the task page should list execution rounds first, then provide entry points to traces and logs. Logs must carry task ID, trace ID, and span ID in their actual context so that after finding a tool anomaly, upstream/downstream lookup is easy. If an external acceptance receipt exists but the trace is missing, the page should state "approval accepted, diagnostic trace unavailable." Only a local submission record without a receipt still requires verification. Missing trace does not prove the business action did not happen. Reliable task and action records handle recovery and verification; traces help explain the process. The two data types can cross-reference but have different retention and purposes.

Test Beyond the Happy Path

First integration test: pause a purchase task, restart the executor, resume with the original task ID. Verify that pre- and post-pause executions are found, and old calls, new calls, and corresponding business actions are distinguished. Next, let budget and contract checks return out of order, mixed with another purchase task's branches. Correct correlation must rely on task and branch IDs, not log timestamp proximity. Simulate duplicate delivery to ensure independent attempts are retained, not merged into one call. Finally, drop a telemetry segment and confirm task records remain queryable, the UI shows the gap explicitly, and an externally accepted but locally timed-out case is traced from the original action to the follow-up query — not mistaken for a new submission.

Short read-only Q&A can start with a single trace; add task-graph correlations only when cross-queue, long-wait, or resume execution become real requirements. Next time a user asks "How's that purchase from yesterday?", start from the task ID — no need to scroll through two days of chat history. Once records are linked, the next article will cover which metrics (latency, cost, outcome) are worth watching.

References

OpenTelemetry: Traces – Span, Event, and Span Links concepts. https://opentelemetry.io/docs/concepts/signals/traces/ OpenTelemetry: Context Propagation – injecting and extracting trace info across processes. https://opentelemetry.io/docs/concepts/context-propagation/ OpenTelemetry: GenAI Semantic Conventions – semantic conventions for model operations; verify version and implementation support when adopting.

https://github.com/open-telemetry/semantic-conventions-genai