Observe Every AI Agent Call Without Changing a Single Line of Code

When an AI Agent receives a user query it typically performs embedding, vector search, rerank, LLM inference and possibly MCP tool calls, spanning multiple providers and protocols. Traditional APM can only report request latency, leaving developers without visibility into which model was used, token consumption, tool calls or retrieval results.

OBI (OpenTelemetry eBPF Instrumentation) solves this by installing a kernel‑level probe that automatically identifies and parses every AI‑related network call, inserting the extracted evidence into OpenTelemetry standard traces and metrics—all without modifying business code.

Step 1: Hook the TLS encryption point. OBI attaches uprobe/uretprobe hooks to the user‑space TLS libraries (OpenSSL/BoringSSL, Go’s crypto/tls, Python’s _ssl) at the moment plaintext buffers are handed to the encryptor (SSL_write) or returned from the decryptor (SSL_read). This captures the clear‑text HTTP request and response without needing private keys or man‑in‑the‑middle certificates.

Step 2: Zero‑copy transfer to user space. The captured buffers are placed directly into a BPF ring buffer (Linux 5.8+). The OBI agent maps the ring buffer into its address space via mmap, reading events with a single copy and using back‑pressure thresholds to avoid impacting the application.

Step 3: Parse HTTP and emit OpenTelemetry spans. In user space the agent reassembles HTTP/1.1 or HTTP/2 frames, selects the appropriate decoder (JSON, SSE, binary), extracts GenAI semantic fields (model, input/output tokens, tool_calls, etc.), enriches the span with Kubernetes metadata, and outputs an OTLP span. The processing pipeline runs as a set of horizontally scalable actors, keeping CPU usage under 1% per node.

OBI’s protocol‑level state machine uses three tiers to reliably identify AI calls: (1) response‑header fingerprints (e.g., OpenAI‑Version, Anthropic‑Organization), (2) URL host and path patterns as fallback, and (3) body‑level key verification (model + messages for LLM, model + input for embedding, etc.). This multi‑level approach avoids false positives even when providers are behind internal gateways.

To reconstruct logical execution contexts across asynchronous runtimes, OBI tracks coroutine and goroutine lifecycles. For Python asyncio it places four uprobes on task creation, step, context copy and execution, mapping each coroutine ID to its parent. For Go it hooks runtime.newproc1 and runtime.casgstatus to build a parent‑child goroutine table, limiting the lookup depth to six levels, which covers 99 % of real‑world call stacks.

Cross‑process trace propagation is achieved by injecting a traceparent header into the plaintext buffer before TLS encryption and reading it back on the receiving side, allowing end‑to‑end tracing across services regardless of language or client library.

Performance measurements on production clusters show a stable ~1 % CPU overhead , thanks to limited uprobe trigger frequency, batch processing in the ring buffer, and user‑space parsing rather than in‑kernel BPF logic.

Real‑world case studies demonstrate OBI’s value: (1) pinpointing a mis‑configured Pinecone namespace that caused irrelevant RAG answers, (2) identifying an internal knowledge‑base agent that sent 80 k‑token prompts and inflated token costs three‑fold, and (3) achieving identical observability for a raw‑HTTP Python agent and an OpenAI‑SDK‑based demo without any code changes.

Future work includes adding a Time‑To‑First‑Token (TTFT) metric for streaming responses, expanding GenAI‑specific metrics (token rate, success/failure per provider/model), full Agent‑level tracing (planning → tool call → observation → response), and multi‑turn conversation correlation.

In summary, OBI turns the operating system into a non‑intrusive recorder for AI Agent activity, delivering OpenTelemetry‑compatible traces and metrics for any language or client implementation, eliminating the observability blind spot that plagues modern GenAI applications.