Building an Observability Platform for LLM Agents with OpenTelemetry

Why an Observability Platform Is Needed

AI agents are becoming the default architecture for production‑grade LLM applications. Compared with a simple API call, an agent involves:

Multi‑step reasoning : plan → execute → reflect → re‑plan

Tool calls : search, compute, database queries, third‑party APIs

Autonomous decisions : conditional branches, retry loops, error recovery

State management : cross‑turn context persistence

These complexities make plain print / logging insufficient.

The Three Pillars of Observability

Trace – full request path (e.g., user query → agent plan → tool call → LLM generate → response)

Metric – overall health trends such as token consumption, P99 latency, tool success rate

Log – detailed discrete events like agent decision logs, exception stacks, audit records

OpenTelemetry Core Architecture

Layered Relationship

OTel initialization follows a "static configuration → dynamic data" layered model.

Key Design Principles

Configuration‑usage separation : a Provider holds static configuration; Tracers are created on demand.

Resource reuse : one Provider can serve multiple Tracers.

Chainable extensions : SpanProcessors can be combined arbitrarily.

Core Components

Resource – describes the emitting entity. Example:

from opentelemetry.sdk.resources import Resource
resource = Resource.create({
    "service.name": "my-agent-service",
    "service.version": "1.0.0",
    "deployment.environment": "production",
})

Each Span inherits these attributes, allowing back‑ends to identify the originating service.

TracerProvider – management hub that creates Tracer instances, manages the SpanProcessor chain, and generates trace_id and span_id:

from opentelemetry.sdk.trace import TracerProvider
provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)

SpanProcessor – processes finished Spans before export. Example using a batch processor:

from opentelemetry.sdk.trace.export import BatchSpanProcessor
processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(processor)

Workflow:

Span ends → queued

When batch size or timeout is reached → sent

Reduces network calls and improves throughput

SpanExporter – sends telemetry to a back‑end such as Jaeger or Prometheus:

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)

Core Concepts of a Span

Full Span Structure (JSON example)

{
    "trace_id": "7bba9e3338b1c1e5d2b6a2b4c3d4e5f6",
    "span_id": "a1b2c3d4e5f67890",
    "parent_span_id": "1234567890abcdef",
    "name": "llm.call",
    "kind": "CLIENT",
    "start_time_unix_nano": 1720615800000000000,
    "end_time_unix_nano": 1720615800500000000,
    "status": {"code": "OK"},
    "attributes": {
        "gen_ai.system": "openai",
        "gen_ai.request.model": "gpt-4o",
        "gen_ai.usage.input_tokens": 150
    },
    "events": [{
        "name": "prompt.sent",
        "timestamp_unix_nano": 1720615800100000000,
        "attributes": {"prompt.length": 150}
    }],
    "links": []
}

Attribute vs Event

Attribute – static metadata of the Span (no timestamp, usually 10‑30 per Span, used to answer "what this operation is"). Queries look like "find Span where model=deepseek".

Event – key moments during the Span’s lifetime (has its own timestamp, usually 0‑5 per Span, used to answer "what happened at this point"). Queries look like "find Span containing prompt.sent".

SpanKind

INTERNAL – internal operation, e.g., agent planning or reasoning steps.

SERVER – receives an external request, e.g., FastAPI entry point or agent service start.

CLIENT – sends an external request, e.g., calling OpenAI API or a search tool.

PRODUCER – produces a message to a queue, e.g., agent pushes a task to Kafka.

CONSUMER – consumes a message from a queue, e.g., a worker processes a task.

Python SDK Manual Instrumentation

Basic Configuration

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "my-agent-service",
    "service.version": "1.0.0",
})
provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)
exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)
tracer = trace.get_tracer("mycompany.agent")

Organising Spans for Agent Workflows

def agent_run(user_query: str):
    # Root span – whole user request
    with tracer.start_as_current_span("agent.run", kind=trace.SpanKind.SERVER) as root:
        root.set_attribute("agent.input", user_query)
        # Planning (internal)
        with tracer.start_as_current_span("agent.plan", kind=trace.SpanKind.INTERNAL):
            plan = generate_plan(user_query)
        # Tool call (client)
        tool_result = call_tool("search", user_query)
        # LLM call (client)
        answer = call_llm(tool_result)
        root.set_attribute("agent.output", answer)
        return answer

def call_tool(tool_name: str, query: str):
    with tracer.start_as_current_span("tool.call", kind=trace.SpanKind.CLIENT) as span:
        span.set_attribute("tool.name", tool_name)
        # ... actual tool invocation ...
        return result

def call_llm(prompt: str):
    with tracer.start_as_current_span("llm.call", kind=trace.SpanKind.CLIENT) as span:
        span.set_attribute("llm.model", "gpt-4o")
        # ... actual LLM invocation ...
        return response

Exception Handling

from opentelemetry.trace import Status, StatusCode
import time

def call_llm_with_retry(prompt: str, max_retries: int = 3):
    for attempt in range(max_retries):
        with tracer.start_as_current_span("llm.call") as span:
            span.set_attribute("retry.attempt", attempt + 1)
            try:
                result = actual_call(prompt)
                span.set_status(Status(StatusCode.OK))
                return result
            except ConnectionError as e:
                span.record_exception(e)
                span.set_status(Status(StatusCode.ERROR, "timeout"))
                if attempt == max_retries - 1:
                    raise
                delay = 2 ** attempt
                span.add_event("retry.scheduled", {"delay_ms": delay * 1000})
                time.sleep(delay)

record_exception()

records exception details. set_status(ERROR) marks the Span as failed for visualisation.

Business‑rule failures set StatusCode.OK with a custom attribute; system errors set StatusCode.ERROR and record the exception.

Automatic Instrumentation and Extensions

Why Automatic Instrumentation?

Manual instrumentation is intrusive, easy to miss, and cannot modify third‑party libraries.

Enabling Automatic Instrumentation

from opentelemetry.instrumentation.openai import OpenAIInstrumentor
from opentelemetry.instrumentation.langchain import LangchainInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXInstrumentor

OpenAIInstrumentor().instrument()
LangchainInstrumentor().instrument()
HTTPXInstrumentor().instrument()

Methods such as chat.completions.create (OpenAI) and AgentExecutor.run (LangChain) now generate Spans automatically.

Extension Modes When Auto‑Instrumentation Misses

Mode 1 – Span Processor (global enhancement)

class BusinessContextProcessor(SpanProcessor):
    def on_start(self, span, parent_context):
        span.set_attribute("deployment.environment", "production")
        from opentelemetry.context import get_value
        user_id = get_value("business.user_id", parent_context)
        if user_id:
            span.set_attribute("business.user_id", user_id)

provider.add_span_processor(BusinessContextProcessor())

Mode 2 – Context Propagation (request‑level attributes)

from opentelemetry.context import set_value, attach, get_current

current = get_current()
current = set_value("business.user_id", user_id, current)
token = attach(current)
# downstream calls automatically inherit this context

Mode 3 – Wrapper Pattern (enhance specific calls)

def call_service_b(data: dict):
    with tracer.start_as_current_span("agent.call_b") as span:
        span.set_attribute("business.request_id", generate_uuid())
        headers = {"Content-Type": "application/json"}
        inject(headers)  # inject trace context
        return requests.post(url, json=data, headers=headers)

Mode 4 – Custom Instrumentor (for libraries without built‑in support)

from opentelemetry.instrumentation.instrumentor import BaseInstrumentor
class MyToolInstrumentor(BaseInstrumentor):
    def _instrument(self, **kwargs):
        # replace original method with instrumented version
        pass

Context Propagation Mechanics

The traceparent header format is illustrated below:

# Service A (caller)
with tracer.start_as_current_span("call_b") as span:
    headers = {}
    inject(headers)  # writes traceparent
    requests.post(url, headers=headers)
# Service B (callee)
carrier = dict(request.headers)
context = extract(carrier)  # extracts trace context
with tracer.start_as_current_span("handle", context=context) as span:
    # parent is Service A's span
    pass

inject()

must be called after start_span to read the active Span. extract() returns a Context, not a Span.

Each downstream call must create fresh headers and inject again; headers cannot be reused.

Collector Deployment and Backend Storage

Why a Collector?

Swap backend: change Collector config instead of modifying every app.

Data sanitisation: centralised handling instead of per‑app implementation.

Sampling control: unified configuration.

Send to multiple backends: Collector duplicates data once instead of each app sending multiple copies.

Collector Configuration (YAML excerpt)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    limit_mib: 512
exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  logging:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [jaeger, logging]

Sampling Strategies

Head sampling – decision at request entry; downstream follows. Simple, reduces traffic volume.

Tail sampling – decide after the Trace completes based on content; keeps errors and slow requests.

Hybrid sampling – head down‑sampling plus tail fine‑grained decision; recommended for production.

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 1000}
      - name: probabilistic
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

Common Distributed‑Environment Issues

Clock Drift

Symptom : Child Span appears before its parent.

Root cause : Unsynchronised NTP/Chrony or VM clock jumps.

Synchronise clocks with NTP/Chrony.

OTel SDK uses a monotonic clock for duration calculations.

Back‑ends (e.g., Jaeger) can correct timestamps and emit warnings.

Key principle : Compare durations, not absolute timestamps.

High‑Cardinality Attributes

Symptom : Using user.id as an Attribute blows up the index.

Root cause : Default inverted index creates one entry per distinct value.

Low‑cardinality data (service, method, status) should stay as Span Attributes – indexable and groupable.

High‑cardinality data (user.id, request.id) should be recorded as Span Events or Logs to avoid index explosion.

High‑Traffic Degradation

# Application side: queue and timeout settings
processor = BatchSpanProcessor(
    exporter,
    max_queue_size=2048,
    schedule_delay_millis=5000,
    export_timeout_millis=3000,
)

# Adaptive sampling based on CPU load
class AdaptiveSampler(Sampler):
    def should_sample(self, *_, **__):
        cpu = psutil.cpu_percent()
        if cpu > 90:
            return TraceIdRatioBased(0.001)  # 0.1% sampling
        elif cpu > 70:
            return TraceIdRatioBased(0.01)   # 1% sampling
        return TraceIdRatioBased(0.1)        # 10% sampling

Agent‑Specific Semantics and Evaluation

Extending OTel Conventions

OTel GenAI conventions already cover gen_ai.system and gen_ai.usage.*. For agents we add custom attributes:

# Agent state
span.set_attribute("mycompany.agent.type", "reAct")
span.set_attribute("mycompany.agent.plan_steps", 3)
span.set_attribute("mycompany.agent.iterations", 2)
# Quality evaluation
span.set_attribute("mycompany.eval.hallucination_score", 0.05)
span.set_attribute("mycompany.eval.relevance_score", 0.92)
# Cost tracking
span.set_attribute("mycompany.cost.input_usd", 0.003)
span.set_attribute("mycompany.cost.total_usd", 0.015)

LLM‑as‑a‑Judge Evaluation

class LLMJudge:
    def evaluate_faithfulness(self, question, context, answer):
        with tracer.start_as_current_span("eval.faithfulness") as span:
            prompt = (
                "Evaluate if the answer is faithful to the context. "
                "Context: " + context + " "
                "Question: " + question + " "
                "Answer: " + answer + " "
                "Return JSON with score(float) and reason(str)"
            )
            response = self.client.chat.completions.create(
                model="gpt-4o",
                messages=[{"role": "user", "content": prompt}],
                temperature=0.0,
            )
            result = json.loads(response.choices[0].message.content)
            span.set_attribute("eval.score", result["score"])
            span.set_attribute("eval.reason", result["reason"])
            return result

Conclusion

First : Get a basic setup working with manual instrumentation and console output.

Then : Automate instrumentation, deploy a Collector, and visualise traces in Jaeger.

Afterwards : Add custom semantics, evaluation systems, and alerting loops.

Finally : Move from "seeing what happened" to "automatically optimising the system".

OpenTelemetry provides the standardised foundation; agent‑specific planning, tool calls, and quality‑assessment semantics must be layered on top to build a true observability platform for LLM agents.