A Kernel‑Embedded ‘Perspective Mirror’: Achieving Full‑Stack Observability with CloudMonitor 2.0 Without Code Changes

Background

In cloud‑native and micro‑service environments, production systems span multiple runtimes (Go, Java, Python, Node.js) and deployment models (containers, Kubernetes, Serverless). Traditional observability requires language‑specific agents or SDKs, forcing code changes, package upgrades, and redeployments for each new service—a cost that grows untenable with rapid development cycles.

AI agents now orchestrate multi‑step LLM workflows, generating complex call chains that span agent orchestration layers, LLM providers, vector databases, and external tools. Conventional APM cannot fully capture these chains, making a zero‑code observability approach essential for AI scenarios as well.

eBPF‑Based Solution: OpenTelemetry eBPF Instrumentation (OBI)

OBI uses Linux kernel eBPF probes to intercept network traffic, library calls, and system calls without modifying application code or restarting processes. It automatically generates OpenTelemetry‑compatible traces and metrics, effectively acting as a "perspective mirror" inside the kernel.

Three Observability Pillars

Application Observability : Distributed tracing (Traces) + RED metrics, covering Web, DB, MQ, GenAI, GPU, etc., with 15+ protocol parsers and automatic trace‑id injection into JSON logs for trace‑log correlation.

Network Observability : L3/L4 traffic monitoring, TCP/UDP statistics, GeoIP, reverse DNS, CIDR tagging, TCP RTT measurement, connection‑failure counts, node‑level global metrics.

Log Enrichment : Language‑agnostic injection of trace_id and span_id into JSON logs, enabling trace‑log correlation.

Protocol Coverage

OBI detects protocols at the kernel level using a three‑stage matching process:

Kernel‑assigned protocol (fastest): Directly reads the event.ProtocolType constant (e.g., MySQL=1, PostgreSQL=2, Kafka=4, etc.).

Deterministic generic matching : Calls detectGenericProtocol which sequentially tries

matchSQL → matchFastCGI → matchMongo → matchCouchbase → matchMemcached

, with SQL parsing that first buffers request then response.

Heuristic fallback : Uses detectHeuristicProtocol to try

matchRedis → matchMemcached → matchHTTP2 → matchNATS → matchAMQP → matchMQTT → matchKafkaFallback

. Ordering is critical (e.g., HTTP/2 must precede MQTT to avoid false positives).

Key anti‑false‑positive details include ASCII prefix filtering for SQL, message‑header validation for PostgreSQL, and frame‑level checks for HTTP/2 vs gRPC.

Deep Language Integration

OBI provides two layers of instrumentation:

Network‑level (language‑agnostic) : All languages obtain basic trace and metrics by intercepting TCP streams.

Runtime‑specific (language‑specific) : uprobe hooks attach to library functions for richer context propagation and trace correlation.

Supported runtimes and integrations:

Go 1.17+: uprobe on 13+ libraries (net/http, gin, gRPC, kafka‑go, sarama, go‑redis, MongoDB, database/sql, etc.) with kernel‑reconstructed goroutine parent‑child relationships.

Java 8+: uprobe on OpenSSL/JVM functions for TLS decryption and cross‑process traceparent propagation.

Node.js 8+: async_hooks uprobe for intra‑process async context propagation.

.NET 6+: OpenSSL uprobe + network‑level kprobe for TLS tracing.

Python 3.9+: asyncio/uvloop uprobe tracking Task and Context objects, rebuilding coroutine parent‑child chains.

Ruby 3.0+: Puma uprobe for queue tracking.

Nginx: dedicated uprobe for HTTP request boundary detection.

Cross‑process traceparent propagation for all non‑Go languages is performed in kernel space via the tpinjector component, using three mechanisms: HTTP/1 header injection, HTTP/2 HPACK injection, and a custom TCP option (kind = 25). Deployment notes warn that some firewalls may strip the custom TCP option.

Data Pipeline Architecture

OBI’s user‑space component is a directed‑acyclic graph (DAG) built from independent agents and an errgroup. The pipeline stages are:

RunWithContextInfo → FindAndInstrument → ReadAndForward → WaitUntilFinished

Key design points:

Swarm orchestration framework : Two‑phase start – instantiate all nodes, then run them concurrently. Failure in any InstanceFunc aborts the whole start.

Message queues ( msg.Queue) provide fan‑out, bypass (zero‑cost short‑circuit), dead‑lock detection, and reference‑counted close semantics.

Ring‑buffer forwarder : Two goroutines (reader and parser) share a pre‑allocated object pool ( poolSize = 2 * BatchLength) to avoid GC pauses. Records are batched (default 100) or flushed on timeout (1 s) and on residual byte checks (every 3 s).

When a span is captured in the kernel, it traverses the DAG, is enriched (e.g., with Kubernetes metadata), filtered, and finally exported to OTEL exporters, printers, or Prometheus metrics.

GPU / CUDA Tracing

OBI also hooks libcuda.so via uprobe to trace kernel launches, graph launches, memory allocations, and async memory copies. Metrics such as cuda_kernel_launch_calls_total, cuda_memory_allocations_total, and cuda_memory_copy_size are exported to Prometheus. The same generic ring‑buffer forwarder processes GPU events, requiring only OTEL_EBPF_CUDA_MODE=auto to enable automatic detection.

Log Enrichment Mechanics

Kernel kprobes on tty_write and pipe_write capture up to 8 KB of log output, retrieve the current trace context, erase the original buffer with bpf_probe_write_user, and send the event via ring‑buffer. The user‑space handler parses JSON logs, injects trace_id / span_id (unless the application already exports OTEL traces), and writes back to the original file descriptor using a path resolver and LRU cache. Known limitations include a small window where logs could be lost if the process crashes between erase and write‑back, and incompatibility with kernels that disable bpf_probe_write_user (e.g., lockdown mode).

Deployment Options

OBI runs as an independent process, Docker container, or Kubernetes DaemonSet on Linux amd64/arm64 kernels (kernel ≥ 5.8, RHEL ≥ 4.18). Feature flags control which of the three pillars are active, and the OTEL_EBPF_BPF_CONTEXT_PROPAGATION variable selects the propagation method (headers, TCP, or disabled).

Observability Use Cases

Detecting anomalous external traffic (e.g., unexpected 2.3 GB/h to a database from a foreign IP) for security alerts.

Identifying cross‑AZ traffic spikes caused by routing changes.

Automatic service‑dependency graph generation for impact analysis.

Network congestion diagnosis via TCP RTT spikes.

Switch or subnet failure detection through sudden connection‑failure spikes.

Integration with CloudMonitor 2.0

Through the CloudMonitor 2.0 console, users can enable “OpenTelemetry Zero‑Code Monitoring” and perform one‑click cluster enrollment. After enrollment, the console displays request counts, error rates, latency, call‑chain traces, network traffic rates, and TCP RTT metrics for each service.

Future work includes extending AI‑Agent observability and adding more network monitoring capabilities.