What's Missing in Agent Observability with Jaeger, Prometheus, Graylog

When building observability for AI agents, the first instinct is to reuse the existing microservice stack: OpenTelemetry for collection, Jaeger for traces, Prometheus with Grafana for metrics, and Graylog for logs. Adding models and tool calls does not mean replacing the entire platform.

The Existing Platform Works, But It Doesn't Know What a Procurement Task Is

Suppose a procurement agent times out submitting an approval. Jaeger shows request latency, Graylog surfaces exceptions, and Grafana may show rising error rates. However, troubleshooting still requires knowing: which procurement item, who confirmed before submission, whether the external system accepted it, and whether a follow‑up audit was triggered.

The collector does not understand these business meanings. Model, retrieval, and tool instrumentation are added by the runtime; authorization checks, waits, retries, and result verifications are supplied by the control plane — the component that manages task state and execution boundaries. A model gateway sees a single model request but may not know the task later entered human review. Business context must be provided by the application itself.

Below is the proposed integration architecture (deployment details still need validation):

flowchart TB
    A[Runtime: Model, Retrieval, Tool Instrumentation] --> C[Collector: Receive, Process, Route]
    B[Control Plane: Checks, Waits, Recovery Events] --> C
    C -->|Traces| J[Jaeger]
    C -->|Metrics Ingestion| P[Prometheus]
    C -->|Compatible Log Protocol| G[Graylog]
    R[Reliable Task, Action & Audit Records] --> X[Export Task Metrics per Agreed Spec]
    X --> P
    P --> F[Grafana: Dashboards & Alert Entry]
    F -.By Scope to Task List.-> V[Task Query Entry]
    R -.Read by Permission.-> V
    J -.Trace Query.-> V
    G -.Log Query.-> V
    E[Business System Receipts & Quality Check Records] -.Verify by Reference.-> V

The Collector receives, processes, and forwards observability data. Solid lines indicate data flow; dashed lines indicate query correlations. Each signal type needs separate receive, process, and export pipelines, and you must verify that the deployed distribution includes the required components. Existing log pipelines can continue directly into Graylog.

Task metrics are aggregated from reliable records. Initially, just add query links to existing task details — no need to build a new console first.

Before Connecting, Confirm What Each Path Receives

OpenTelemetry is an observability standard and toolkit; OTLP is its transport protocol. Even when the protocol name matches, you must verify signal type, transport method, authentication, and field mapping.

Before integration, check each destination:

Traces to Jaeger : Write via its OTLP receiver. Verify deployed version, receiver configuration, and storage backend. (Source: Jaeger Architecture)

Metrics to Prometheus : Keep pull‑based scraping, or enable OTLP/HTTP receive if supported. Do not assume OTLP/HTTP is enabled by default, and do not send gRPC data to an HTTP endpoint. Choose one primary ingestion path per metric set to avoid double counting. (Sources: OpenTelemetry Prometheus Exporter, Prometheus OTLP Guide)

Logs to Graylog : Retain existing log pipeline, or use Graylog's OpenTelemetry input. In Graylog 7.1, the OpenTelemetry input accepts OTLP/gRPC logs only — not metrics, traces, or OTLP/HTTP. If the input is missing, stick with the proven log collector and add correlation fields; do not force a protocol upgrade. (Source: Graylog OpenTelemetry Input)

Dashboards to Task Details : Enter task list by type and time range, then drill down to individual tasks. Verify query conditions, data freshness, and access control.

Field mapping is a common pitfall. The Graylog input maps trace_id to otel_trace_id, and attributes get corresponding prefixes and naming conversions. If the query UI still searches by the original field name, data appears missing even though it was ingested.

Therefore, acceptance testing must send a real test record and retrieve it from the query side, checking identifiers, timestamps, versions, and results. Seeing the collector start successfully does not prove data arrived.

Add an Entry Point So People Can Find Evidence from Tasks

Returning to the procurement timeout example: after opening the task detail, the user should first see "Submission result pending verification" and the latest query time, not dozens of model calls.

Engineers can then drill down to the original action, confirmation basis, submission call, exception logs, and recovery query results. If an external receipt was obtained, show it; if still unknown, retain the unknown state — do not let a model summary replace it.

This entry reads reliable records and provides links to Jaeger, Graylog, and quality‑check results. It only handles queries; it does not infer or rewrite task state from diagnostic logs. If caching is used, display the last update time so stale data does not masquerade as real‑time.

Navigation from summary dashboards to specific tasks also needs design. Regular metric labels don't carry every task ID, so first jump to the list for the relevant time window and task type; if a trace sample references a task, navigate directly. Without a sample, a query entry must still exist.

Cross‑page jumps require authorization. The target system or controlled query service must validate task ownership and access scope — not rely solely on tenant parameters in the URL.

Decide What to Collect Before Deciding How Long to Store

Agent debugging easily slides into "record all inputs and outputs." Short‑term debugging convenience can long‑term replicate contracts, personal data, and credentials across multiple systems.

The chosen default is to record task correlations, operation names, durations, results, usage, versions, and evidence references. Prompts (instructions and inputs to the model) and full responses, plus tool parameters, are saved only when troubleshooting demands it.

Sensitive content should be blocked at the source; Collector filtering and redaction add a second check. Simply stripping a password field is insufficient — contracts and tool response bodies may also contain sensitive data.

When original content must be inspected, prefer controlled references with permission‑based access. Detailed content can have a shorter retention period, but key business evidence must be kept per business requirements; references must remain resolvable to the correct version for the required duration.

Defaulting to no full‑text retention makes some intermittent issues hard to fully reproduce. When temporary collection is enabled, limit it by task scope, time window, and authorized personnel, and estimate daily volume and storage budget to prevent debug switches from staying on indefinitely.

When Collection System Fails, Business Must Not Be Dragged Down

Ordinary diagnostic data suits async batched reporting, but application‑side queues, Collector buffers, and retries must have upper bounds. When downstream is unavailable, unlimited accumulation cannot guarantee "zero loss."

OpenTelemetry's reliability documentation explicitly states that queue saturation and retry timeouts can still lose data; persistent queues improve recovery under some failures but are not infinite capacity guarantees. (Source: Collector Resiliency)

What gets dropped when queues fill must be agreed in advance. Dropped counts, export failures, and backlogs must be visible. However, critical authorizations, submission intents, and receipts must be reliably persisted per task and audit requirements. If a high‑risk operation requires an audit record first, a write failure should halt or escalate to manual handling per agreement — not be discarded alongside routine diagnostic logs.

Sampling has boundaries. Tail‑based sampling decides after collecting a trace segment whether to keep it, favoring errors or slow calls, but is limited by waiting windows, memory, and data completeness. Upstream‑lost segments cannot be recovered; a task issue surfacing the next day may not retrieve the previous day's full trace. (Source: OpenTelemetry Sampling)

Collection latency, drop rates, query availability, and missing correlations must be monitored. Separate statistics for policy‑based sampling vs. anomalous loss. Probe collection and query entry points independently. Fault alerts need a separate working channel; otherwise a collector outage cannot even report its own failure.

UIs should distinguish "no anomaly recorded" from "temporarily unable to fetch records," showing the last successful collection or query time.

Connect One Task First, Then Consider Scaling

Start with a single procurement task to validate the integration path. Acceptance focuses on four steps:

Submit a test task; from the task detail locate model and tool calls, logs, and business receipts; verify identifiers and versions.

Pause the task and restart the executor; confirm before/after records still correlate to the same task, and alerts route the responsible person to the correct entry point.

Disconnect the collector or downstream storage; check business latency, queue limits, and independent alerts; after recovery, verify replay, duplicates, and missing records. Validate execution boundaries for critical audit write failures separately.

Inject fictitious sensitive content; query as an unauthorized user; verify collector filtering, raw‑content access control, and cross‑tenant isolation.

Acceptance also clarifies ownership: platform team owns shared pipeline issues; application team owns instrumentation gaps; task owner confirms whether state and evidence are sufficient.

Specialized Agent observability products can add execution traces, prompt versioning, or evaluation management. Adoption depends on whether they integrate with existing data and permission systems, and how much storage, migration, and maintenance cost they add — not just on UI polish.

At this point, a procurement task provides a troubleshooting entry. But even with full traces, another problem may emerge: the process looks fine, yet the answer is wrong. The next article will cover how quality evaluation leverages these records.