Build a Production-Ready Observability Platform with OpenTelemetry

Background and Motivation

The article starts with a real incident where a payment API’s P99 latency spikes from 80 ms to 3.2 s without a corresponding error rate increase. Traditional tools (Prometheus/Grafana, ELK/Loki, Jaeger/Zipkin/Tempo) each provide isolated views—metrics show latency, traces show call paths, logs show events—forcing engineers to manually stitch information together. The root cause turned out to be a thread‑pool queue in the payment service, not Redis or the database. This illustrates the need for a unified observability platform that can correlate trace, metric, and log data through shared identifiers.

Problems with Conventional Monitoring Stacks

Three major issues are identified:

Duplicate collection pipelines for each signal type.

Inconsistent field semantics across services (e.g., pay-service vs payment vs payment-service).

Context fragmentation—logs lack trace_id, metrics lack exemplars, and traces miss business dimensions.

Additional operational pain points include difficulty governing fields, high storage cost, and scaling challenges as the number of services and clusters grows.

OpenTelemetry’s Positioning

OpenTelemetry is presented not as a replacement for existing tools but as a standard that unifies data collection, processing, and export. Its architecture consists of an API/SDK/Java Agent layer that produces Trace ( Span), Metric ( Counter, Histogram, Gauge, UpDownCounter), and Log ( LogRecord) objects, which are sent via the OTLP protocol to an OpenTelemetry Collector. The collector can enrich, filter, sample, batch, and route data to back‑ends such as Tempo, Prometheus/Mimir/VictoriaMetrics, and Loki/Elasticsearch.

Core Data Model

OpenTelemetry defines a three‑layer model:

Resource
 └─ InstrumentationScope
      ├─ Span
      ├─ Metric DataPoint
      └─ LogRecord

Key Resource attributes (service name, namespace, version, environment, Kubernetes namespace/pod, cloud region) must be consistent across all signals. The article provides a table of example fields and stresses that service.name must be unique and stable.

Signal Details

Trace

Spans contain identifiers ( trace_id, span_id, parent_span_id), kind, attributes, events, and status. Context propagation follows the W3C Trace Context standard (e.g., traceparent header). Example trace hierarchy for an order creation request is shown.

Metric

Metrics are categorized into infrastructure, runtime, and business layers. The article recommends using Histogram for latency and amount distributions (P95, P99, max). It also lists concrete metric types (Counter, Histogram, Gauge, UpDownCounter) with example names.

Log

Structured JSON logs are advocated. A sample log record includes trace_id and span_id fields, enabling one‑click navigation from logs to traces.

Sampling Strategies

Two approaches are compared:

Head Sampling – decision made at the SDK/agent; low overhead but cannot react to downstream errors.

Tail Sampling – decision made in the collector after the full trace is observed; higher cost but can retain error or slow‑request traces.

Recommended production configuration uses a hybrid sampler: ParentBased + TraceIdRatioBased at the SDK (e.g., 10 % sampling) and Tail Sampling policies in the collector to keep 100 % error and slow traces.

Collector Architecture and Deployment

The recommended topology for medium‑to‑large Java microservice environments is "Agent → Gateway Collector → Kafka → Backend Stores". The gateway collector performs enrichment, attribute normalization, sanitization (deleting Authorization, Cookie, DB statements), and tail sampling before routing data to:

Tempo (trace storage) via Kafka.

Prometheus/Mimir/VictoriaMetrics (metrics) via remote write.

Loki/Elasticsearch (logs) via OTLP HTTP.

Deployment modes (Sidecar, DaemonSet, Gateway, Hybrid) are compared, with the gateway mode recommended for centralized policy management and scalability.

Java Service Integration

Automatic Instrumentation

The OpenTelemetry Java Agent automatically instruments HTTP, JDBC, Redis, RPC, JVM, and thread pools. Manual instrumentation is reserved for critical business spans and custom metrics.

Configuration Example

java \
  -javaagent:/opt/otel/opentelemetry-javaagent.jar \
  -Dotel.service.name=order-service \
  -Dotel.resource.attributes=service.namespace=mall,service.version=1.8.3,deployment.environment.name=prod,k8s.cluster.name=prod-a \
  -Dotel.exporter.otlp.endpoint=http://otel-collector-gateway:4317 \
  -Dotel.exporter.otlp.protocol=grpc \
  -Dotel.traces.exporter=otlp \
  -Dotel.metrics.exporter=otlp \
  -Dotel.logs.exporter=none \
  -Dotel.traces.sampler=parentbased_traceidratio \
  -Dotel.traces.sampler.arg=0.1 \
  -Dotel.instrumentation.runtime-telemetry.enabled=true \
  -jar app.jar

The agent injects trace_id and span_id into the logging MDC, allowing JSON logback configurations to emit these fields automatically.

Sample Business Code

A complete order‑creation flow is presented, showing how to create a root span, record attributes (order channel, payment method), emit metrics ( business.order.created, business.order.amount), handle errors with recordException, and publish events. The code also demonstrates manual spans for inventory checks, payment confirmation, and order persistence, each setting relevant attributes and status codes.

Async Context Propagation

Incorrect usage (losing context) and the correct pattern using Context.current() or a TraceContextAwareExecutor wrapper are shown.

Kafka Propagation

When the Java Agent does not automatically propagate trace context, the article provides a manual example using TextMapPropagator to inject/extract traceparent headers on producer and consumer records.

Logback JSON Configuration

<configuration>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <fieldNames>
                <timestamp>timestamp</timestamp>
                <level>level</level>
                <logger>logger</logger>
                <thread>thread</thread>
                <message>message</message>
            </fieldNames>
            <includeMdc>true</includeMdc>
            <customFields>{"log.format":"json"}</customFields>
        </encoder>
    </appender>
    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
    </root>
</configuration>

The resulting JSON log includes trace_id, span_id, and trace_flags fields.

OpenTelemetry Collector Configuration (Gateway)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 2048
    spike_limit_mib: 512
  k8sattributes:
    auth_type: serviceAccount
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.deployment.name
        - k8s.node.name
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.ip
      - sources:
          - from: connection
  resource:
    attributes:
      - key: telemetry.sdk.provider
        value: opentelemetry
        action: upsert
      - key: observability.platform
        value: unified-otel
        action: upsert
  attributes/sanitize:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: http.request.header.cookie
        action: delete
      - key: http.response.header.set_cookie
        action: delete
      - key: db.statement
        action: hash
  filter/drop_noise:
    error_mode: ignore
    traces:
      span:
        - 'attributes["url.path"] == "/actuator/health"'
        - 'attributes["http.route"] == "/actuator/health"'
    metrics:
      metric:
        - 'name == "process.runtime.jvm.buffer.limit"'
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 5000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-requests
        type: latency
        latency:
          threshold_ms: 800
      - name: important-services
        type: string_attribute
        string_attribute:
          key: service.name
          values:
            - payment-service
            - order-service
          enabled_regex_matching: false
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

exporters:
  kafka/traces:
    brokers:
      - kafka-0.kafka:9092
      - kafka-1.kafka:9092
      - kafka-2.kafka:9092
    topic: otel-traces
    encoding: otlp_proto
    protocol_version: 3.6.0
    compression: gzip
  prometheusremotewrite:
    endpoint: http://mimir-nginx.observability.svc:9009/api/v1/push
    resource_to_telemetry_conversion:
      enabled: true
  otlphttp/loki:
    endpoint: http://loki-gateway.observability.svc:3100/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, resource, attributes/sanitize, filter/drop_noise, tail_sampling, batch]
      exporters: [kafka/traces]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, resource, filter/drop_noise, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, resource, attributes/sanitize, batch]
      exporters: [otlphttp/loki]

The configuration includes memory limiting, Kubernetes attribute enrichment, field sanitization, noise filtering, tail sampling policies, and batch processing.

Kafka as a Buffer for Traces and Logs

Traces are sent to a Kafka topic (e.g., otel-traces) and consumed by a separate collector that writes to Tempo. Metrics are typically sent via remote write directly to Mimir or VictoriaMetrics, avoiding Kafka because they are pull‑based.

High‑Concurrency Design Considerations

The article lists failure points under load (SDK queue overflow, collector CPU spikes, tail‑sampling cache growth, insufficient Kafka partitions, back‑end write throttling, high‑cardinality labels). Recommendations include tuning SDK sampling rates, batch queue sizes, export batch sizes, using gRPC + gzip, and configuring Collector memory limits.

Scaling the Collector

A stateless Gateway Collector should be deployed with at least three replicas and autoscaled based on CPU and memory utilization. Important Collector metrics to monitor are otelcol_receiver_accepted_spans, otelcol_exporter_send_failed_spans, and otelcol_processor_dropped_spans. Tail sampling requires careful load‑balancing to keep a single trace on the same collector.

Kafka Topic Design

Separate topics per signal and environment (e.g., otel-prod-traces, otel-prod-logs) are recommended. Partition counts are sized based on expected span volume (12 partitions for < 1 billion spans, 24‑48 for 1‑10 billion, 64+ for > 10 billion).

Dashboards, Alerts, and SLOs

Standard RED (Rate, Errors, Duration) dashboards are built with PromQL queries. Example queries for request rate, error rate, and P99 latency are provided. TraceQL examples show how to find slow or error traces. LogQL queries demonstrate jumping from a trace ID to related logs. Alert rules for high error rate and high P99 latency are defined in Prometheus alerting format.

Operational Best‑Practice Checklist

Enforce consistent service.name, service.version, and environment attributes.

Ensure logs contain trace_id / span_id via MDC.

Implement RED metrics for every service.

Define business‑level metrics (order created, payment confirmed, etc.).

Configure sampling (head + tail) and sanitization at the collector.

Avoid high‑cardinality labels in metrics; keep them in traces only.

Monitor Collector health and enable HPA.

Roadmap

The rollout is split into four phases: (1) unify trace IDs and service naming; (2) add metrics and alerts; (3) introduce sampling, data‑masking, and cost governance; (4) scale to multi‑cluster and add intelligent anomaly detection.

Conclusion

OpenTelemetry transforms observability from a collection of siloed tools into a unified data model that enables trace‑to‑log correlation, efficient root‑cause analysis, and scalable operations. By following the detailed architecture, code samples, and governance practices, teams can dramatically improve fault‑diagnosis speed and maintain a cost‑effective, production‑grade observability platform.

References

OpenTelemetry Java documentation – https://opentelemetry.io/docs/languages/java/

OpenTelemetry Semantic Conventions – https://opentelemetry.io/docs/specs/semconv/

Grafana Tempo TraceQL – https://grafana.com/docs/tempo/latest/traceql/

OpenTelemetry Kubernetes Operator – https://opentelemetry.io/docs/platforms/kubernetes/operator/automatic/