OpenTelemetry in Production: Unified Observability Collection with Auto-Instrumentation, Collector & Tail Sampling

Why Unified Collection Is Needed

Many teams currently stitch together three separate observability pillars: Traces via Jaeger client or language-specific SDKs, Metrics via Prometheus client libraries or Micrometer, and Logs via logging frameworks scraped by Fluent Bit or Promtail. This approach creates four major problems:

SDK proliferation : A Java service must pull in Jaeger client, Micrometer, SLF4J, and a log agent. Version conflicts make troubleshooting take longer than writing business code.

Vendor lock-in : Data formats are proprietary. Switching from Jaeger to Tempo requires changing export configuration in every service.

Three pipelines, three ops burdens : Collector, Prometheus, and log agents each need separate deployment, monitoring, and alerting. Complexity grows multiplicatively.

Missing correlation : Trace IDs and log trace_ids are generated independently, so traces and logs cannot be linked (the root cause behind the three-pillar correlation challenge).

OpenTelemetry (OTel) standardizes the collection layer : one API/SDK, one transport protocol (OTLP), one Collector pipeline, feeding any backend. OTel does not replace Prometheus, Loki, or Tempo; it unifies their collection frontend . Backends remain interchangeable, but the SDK and pipeline are maintained once.

Application (instrumentation) ──OTLP──▶ Collector (receive/process/export) ──▶ Tempo / Prometheus / Loki
  SDK/auto-injection                    receivers→processors→exporters        Any backend

What OTel Is: Three Components + One Protocol

Hosted by CNCF and graduated, OTel is the de facto standard for cloud-native observability. Its components:

Specification : Defines API, data model, and semantic conventions (field naming).

API/SDK : Per-language libraries that generate telemetry data.

Collector : Pluggable collection/processing/export pipeline, the heart of OTel.

OTLP (OpenTelemetry Protocol) : Unified transport protocol over gRPC and HTTP.

Key principle: OTel standardizes generation and transport , not storage or visualization . It is a backend-agnostic front layer.

Architecture: Instrumentation → OTLP → Collector → Backend

3.1 Instrumentation (Data Collection)

Two approaches, neither requiring business code changes:

Auto-injection (recommended for start) : OTel Operator uses Kubernetes Mutating Webhook to inject instrumentation agents at Pod creation. Developers only add an annotation.

Manual SDK (fine-grained control) : Import OTel SDK in code to manually instrument critical paths (custom spans, business attributes).

Most teams achieve 90% coverage with auto-injection; the remaining 10% of key business logic uses manual SDK.

3.2 Transport Protocol (OTLP)

All signals use OTLP on two default ports:

4317 : gRPC (default, better performance).

4318 : HTTP/Protobuf (fallback when firewalls block gRPC).

OTLP carries traces, metrics, and logs simultaneously — the physical basis for unification.

3.3 Collector Deployment: Agent + Gateway Pattern

┌─────────────┐   ┌─────────────┐   ┌──────────────┐   ┌────────┐
│ Pod (Node)  │   │  Agent      │   │  Gateway     │   │ Tempo  │
│ app + SDK   │──▶│ Collector   │──▶│ Collector    │──▶│ Prom   │
│             │   │ (DaemonSet) │   │ (Deployment) │──▶│ Loki   │
└─────────────┘   └─────────────┘   └──────────────┘   └────────┘

Agent (DaemonSet per node) : Receives locally, enriches with resource attributes, batches, forwards to Gateway.

Gateway (central Deployment) : Aggregates, makes sampling decisions, exports to backends, handles unified auth.

Why two layers? Agent guarantees local collection even if Gateway fails (caches batches). Gateway centralizes sampling and routing. Small clusters can run Gateway only; Agent is optional.

Zero-Code Onboarding: OTel Operator Auto-Injection

Lowest-friction path — developers change zero code.

Step 1: Deploy OTel Operator

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Step 2: Define Instrumentation (global collection config, written once)

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: otel-java-instrumentation
  namespace: observability
spec:
  exporter:
    endpoint: http://otel-gateway.observability:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.5"   # 50% head sampling (start conservative, tail sampling backs it up)
  java:
    env:
      - name: OTEL_TRACES_SAMPLER
        value: "parentbased_traceidratio"

Step 3: Add Annotation to Business Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  template:
    metadata:
      annotations:
        # Reference the Instrumentation above, auto-inject Java agent
        instrumentation.opentelemetry.io/injected-java: "otel-java-instrumentation"
    spec:
      containers:
        - name: payment-api
          image: registry/payment-api:1.4.2

At Pod creation, the Operator's Webhook automatically:

Injects OTel Java agent init container and JVM arguments.

Sets OTEL_* environment variables.

Application produces traces + metrics with zero code changes.

Auto-Injection Annotations by Language

Java: instrumentation.opentelemetry.io/injected-java Python: instrumentation.opentelemetry.io/injected-python Node.js: instrumentation.opentelemetry.io/injected-nodejs.NET: instrumentation.opentelemetry.io/injected-dotnet Go: Auto-injection not yet supported (requires manual SDK).

Pure Environment Variable Injection (No Operator, Bare Metal / Non-K8s)

# Java: attach javaagent
java -javaagent:/opt/opentelemetry-javaagent.jar \
  -Dotel.service.name=payment-api \
  -Dotel.exporter.otlp.endpoint=http://otel-gateway:4317 \
  -jar app.jar

# Python: wrap with opentelemetry-instrument
opentelemetry-instrument \
  --traces_exporter otlp \
  --metrics_exporter otlp \
  --service_name payment-api \
  python app.py

# Node.js: register auto-instrumentations
NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register" \
  OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-gateway:4317 \
  OTEL_SERVICE_NAME=payment-api \
  node app.js

Collector Configuration in Practice

Collector config is the core of OTel. A production-ready Gateway config skeleton:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  # Legacy Jaeger compatibility (migration period)
  jaeger:
    protocols:
      grpc: {}
      thrift_http: {}
  # Legacy Prometheus scrape compatibility (migration period)
  prometheus:
    config:
      scrape_configs:
        - job_name: otel-collector
          static_configs:
            - targets: [0.0.0.0:8888]

processors:
  # MUST BE FIRST: memory limiter to prevent Collector OOM (core stability)
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 500

  # Batching: accumulate before sending, drastically reduces backend pressure (core performance)
  batch:
    timeout: 5s
    send_batch_size: 1024

  # K8s attribute enrichment: auto-add namespace/pod/node/deployment info
  k8sattributes:
    auth_type: serviceAccount
    extract:
      metadata:
        - namespace
        - pod_name
        - deployment

  # Resource attributes: add global labels like cluster, environment
  resource:
    attributes:
      - key: deployment.environment
        value: prod
        action: insert
      - key: k8s.cluster.name
        value: prod-cluster-01
        action: insert

  # Tail sampling (traces only): keep errors & slow requests, sample rest (core cost control)
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 500
      - name: probabilistic
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

exporters:
  # Traces → Tempo (via OTLP/HTTP)
  otlphttp/tempo:
    endpoint: http://tempo:4318
  # Metrics → Prometheus (expose scrape endpoint, reuse existing Prometheus)
  prometheus:
    endpoint: 0.0.0.0:8889
    resource_to_telemetry_conversion:
      enabled: true
  # Logs → Loki
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
  # Debug: print to stdout
  debug:
    verbosity: normal

service:
  pipelines:
    traces:
      receivers: [otlp, jaeger]
      processors: [memory_limiter, k8sattributes, tail_sampling, batch]
      exporters: [otlphttp/tempo, debug]
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, k8sattributes, resource, batch]
      exporters: [prometheus, debug]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, resource, batch]
      exporters: [loki, debug]

Critical Processor Ordering Discipline

memory_limiter

must be first in processors list — it must intercept memory pressure early. batch goes last (just before export) to batch processed data. tail_sampling only in traces pipeline, and must precede batch (sample then batch).

Metrics do not use tail_sampling (sampling breaks metric aggregation); use recorder-side or head sampling instead.

Sampling Strategy: Core of Cost Control

Collecting 100% of traces at millions of QPS blows up storage and backend costs. Sampling is not optional optimization; it is mandatory.

Two-Level Sampling Combination

Head sampling (SDK side) : parentbased_traceidratio at 50%. Drops half at ingress, reducing collection and transport load.

Tail sampling (Gateway side) : Errors 100% + slow requests 100% + remainder 10%. Guarantees valuable traces are kept.

Why both? Head sampling cuts volume early but is blind — it doesn't know if a trace will error. Tail sampling sees full result (failure/latency) but already paid collection cost. Together: head sampling reduces volume, tail sampling preserves quality.

Tail Sampling Config Key Points (from config above)

decision_wait: 10s</strong>: Wait 10 seconds for all spans of a trace to arrive before deciding, avoiding broken traces where parent unsampled but child sampled.</li><li>Errors and slow requests kept entirely — these are what you most need to see.</li><li>Probabilistic 10% fallback retains enough samples for overall shape.</li></ul><p>Metrics and logs generally not sampled (or extremely light): metrics are aggregated values, dropping samples causes distortion; logs filtered by level (e.g., drop below INFO) is more sensible than sampling.</p><h2>Integrating Existing Systems: Migrating from Jaeger / Prometheus</h2><p>OTel receivers are designed for painless takeover of legacy systems. Migration can run in parallel, then cut over gradually:</p><ul><li><strong>Jaeger client</strong> → <code>jaeger

receiver: Point app's Jaeger exporter to Collector's Jaeger port.

Zipkin → zipkin receiver: Point to Collector's Zipkin port.

Prometheus scrape → prometheus receiver: Move scrape config into Collector, or keep Prometheus scraping directly.

Custom OTLP → otlp receiver: Direct connect.

Migration Phases

Phase 1 : Bring up Collector with otlp + jaeger + zipkin receivers, export to both Tempo (new) and old Jaeger (old). Both have data; verify consistency.

Phase 2 : Switch app trace exporters from direct Jaeger to Collector. Old Jaeger receives no new writes.

Phase 3 : Decommission old Jaeger; Collector becomes sole trace entry point.

Handling existing Prometheus : No rush. Prometheus continues scraping RED/USE metrics. OTel only takes over application active instrumentation and trace/log. When ready, use prometheus receiver to bring scrape jobs into Collector. Don't unify for unity's sake — stability first .

Log Collection: filelog + Trace Correlation

OTel can also unify log collection (the Loki logging chapter covers storage; here we swap the collection front-end to OTel).

Use filelog Receiver to Scan Container Logs

receivers:
  filelog:
    include: [/var/log/pods/*/*/*.log]
    start_at: beginning
    operators:
      - type: docker
        stream: stdout
      - type: regex
        regex: '^(?P<ts>\S+) (?P<level>\w+) (?P<msg>.*)$'
      - type: traceparser   # Extract trace_id from logs to enable correlation
        trace_id:
          regex: 'trace_id["\s:=]+([0-9a-f]{32})'

exporters:
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [loki]

Key value : The traceparser operator extracts trace_id from logs, writes it as a Loki label. When viewing a slow trace in Tempo, you can jump directly to all logs for that request — the three-pillar correlation foundation.

Pitfall Avoidance Guide

Pitfall 1: Missing memory_limiter → Collector OOM

Symptom : Traffic spike causes Collector memory surge, K8s kills it, entire collection pipeline goes down — worse than no monitoring.

Fix : memory_limiter must be first processor; set limit_mib with 20% headroom under Pod memory limit. This is the #1 stability switch for OTel deployments.

Pitfall 2: Skipping batch → Backend Drowned in Tiny Packets

Symptom : Each span exported individually; Tempo/Prometheus connection pools saturated, processing latency spikes.

Fix : batch processor mandatory, send_batch_size 1024, timeout 5s. Batching reduces backend pressure by an order of magnitude.

Pitfall 3: 100% Sampling → Cost Explosion

Symptom : Trace volume grows linearly with traffic; first month's bill manageable, traffic doubles and it collapses.

Fix : Both head sampling (SDK) and tail sampling (Gateway) required. Keep all errors/slow requests, sample normal traffic 10-50%. Logs filtered by level, not sampling.

Pitfall 4: OTLP Version Mismatch → Silent Data Loss

Symptom : Mixed OTLP protocol versions across Collector/SDK versions cause field drops or entire trace loss with no obvious errors.

Fix : Pin all OTel components to same release line; upgrade Collector before SDK; use debug exporter to spot-check field completeness.

Pitfall 5: Wrong Auto-Injection Annotation → Pod Not Instrumented

Symptom : Annotation added but no effect; traces empty.

Fix : Ensure annotation is on spec.template.metadata.annotations (pod template, not deployment metadata); verify Operator webhook ready; check Pod has extra init container (sign of successful injection).

Pitfall 6: Duplicate Log Collection

Symptom : Same container logs scanned by OTel filelog and Fluent Bit/Promtail; duplicates in Loki.

Fix : When migrating logs to OTel, disable original log agents; use single collection entry point to avoid two pipelines fighting over same files.

Closing Thoughts

OpenTelemetry's value goes beyond "fewer SDKs" — it standardizes the observability collection frontend . The ripple effects:

Backend freedom : Today Tempo+Prometheus+Loki, tomorrow something else — collection layer unchanged.

Three pillars connected : Unified protocol lets trace, metric, log share semantic conventions and attributes; correlation becomes natural (the "three-pillar correlation" finally has ground).

Cost control : Collector centralizes sampling and batching, ending per-backend billing chaos.

This article completes a trilogy with previous posts: Prometheus vs VictoriaMetrics (metrics storage selection), ELK→Loki Migration (log storage selection), and this one solving the first kilometer of "how to collect uniformly". Together they form a complete, open, vendor-lock-free observability stack.

Start today: Deploy a Collector in test, add that one annotation to a Java service, watch traces flow into Tempo. Ten minutes and you've touched the OTel threshold.