Spring Boot Observability Trio: Metrics, Tracing, Logging Trade-offs & Integration

Problem Definition: Three Questions, Three Signals

Observability tools answer three distinct granularity questions in sequence: Metrics answer "Is the system healthy?" (aggregated view), Tracing answers "Which segment is slow/error?" (single-request view), Logging answers "What exactly happened?" (event details). The critical insight: these three must correlate on the same request via a shared traceId; otherwise teams manually join data across tools.

Each signal has a different cost structure: Metrics are small but suffer from cardinality explosion (e.g., adding userId as a label creates millions of time series); Tracing cost scales with request volume (sampling required at scale); Logging is unbounded and can spike 100x during error storms. No signal is "more is better" — governance is essential.

Technical Analysis: Spring Boot 3.x Unified Instrumentation

Spring Boot 3.2/3.3 introduces the Observation API (Micrometer 1.12+, micrometer-tracing 1.2+) as a single instrumentation entry point. One Observation simultaneously emits a Metric (Timer histogram) and a Span (trace segment), eliminating the 2.x need for dual annotations ( @Timed + @NewSpan).

Observation.createNotStarted("order.create", registry)
  .lowCardinalityKeyValue("channel", "app")
  .highCardinalityKeyValue("order.no", cmd.orderNo())
  .observe(() -> orderService.create(cmd));
// Produces both Metrics and Span

Key distinction : lowCardinalityKeyValue goes to both Metrics and Tracing; highCardinalityKeyValue goes only to Tracing. Putting userId in lowCardinality causes cardinality explosion.

Bridge Selection: OTel vs Brave

Micrometer-tracing supports two bridges: bridge-otel (OpenTelemetry) and bridge-brave (Zipkin/B3). The article recommends OTel bridge for new projects because it standardizes propagation (W3C), sampling, and export protocol (OTLP), decoupling application from backend. Choose Brave only if deeply invested in Zipkin ecosystem with no migration plans.

bridge-otel : larger dependency size; W3C/B3 configurable propagation; full Baggage support; OTel ecosystem; choose for standardization.

bridge-brave : smaller dependency; B3 propagation; limited Baggage support; Zipkin ecosystem; choose only if existing deep Zipkin customization with no migration plans.

Backend Stack

For small/medium teams: Prometheus + Loki + Tempo (or Jaeger) + Grafana covers 90% of needs with native traceId cross-linking. Commercial APM considered only when self-hosted stack becomes unmaintainable; OTLP ensures low switching cost.

Core Request Flow: TraceId Propagation & MDC

An HTTP request passes through ObservationFilter which extracts/injects W3C traceparent header, starts an Observation, and places traceId/spanId into MDC (Mapped Diagnostic Context) — a thread-local Map. Logback pattern %X{traceId}/%X{spanId} renders them in logs. Critical : MDC is ThreadLocal; async boundaries (@Async, thread pools, CompletableFuture, Reactor) lose context unless wrapped with TaskDecorator or context-propagation library.

# W3C traceparent format
# Inbound: 00-4bf92f...-00f067...-01
# Outbound: 00-4bf92f...-7c3f2a...-01 (new spanId, same traceId)

Source Code Deep Dive: Five Pivotal Classes

ObservationRegistry : Broadcasts lifecycle events (start/stop/error) to registered handlers (MetricsHandler, TracingHandler) — a publish-subscribe model enabling custom instrumentation.

ObservationThreadLocalAccessor : Stores current Observation scope in a ThreadLocal; Reactor's context-propagation library moves it across threads only if thread pools are wrapped.

DefaultMeterRegistry (PrometheusMeterRegistry) : Performs client-side aggregation; each label combination becomes an independent time series in Prometheus, explaining why cardinality must be bounded.

Sampler : Decides at span creation whether to record. Default respects parent sampling decision (head-based). Tail-based sampling (keep errors 100%) requires collector-level support, not application-level.

LogbackEncoder (LoggingEventCompositeJsonEncoder) : Emits structured JSON logs with traceId/spanId as queryable fields, enabling "jump from trace to logs". Plain text logs cannot do this reliably.

Metrics Implementation: Types, Naming, Cardinality Guardrails

Metric Types

Counter : Monotonic counts (sum(rate()))

Gauge : Instantaneous values (connection pool size)

Timer : Latency distribution + count (histogram quantiles) — always use for latency

DistributionSummary : Non-time sizes (payload bytes)

Common mistake: using Counter for latency loses distribution (P99 invisible).

Naming & Label Conventions

Metric name: domain.object.action with unit suffix (e.g., order.create.duration, cache.hit.total). Iron rule : label values must be a bounded set (uri, method, status, channel). userId, orderId, request params are forbidden — they belong in logs/traces.

Cardinality Governance

Real incident: http_requests_total with uri(200) × method(5) × status(8) = 8,000 series (fine). Adding userId(100k) → 800M series → Prometheus OOM. Mitigation: pre-deployment cardinality review; MeterFilter to enforce limits and deny high-cardinality labels.

@Bean
MeterFilter limitFilter() {
  return MeterFilter.maximumAllowableTags("http", "uri", 100, MeterFilter.deny());
}
@Bean
MeterFilter denyHighCard() {
  return MeterFilter.deny(id -> id.getTag("userId") != null);
}

Tracing Implementation: Sampling, Baggage, Custom Spans

Minimal Setup

dependencies:
  - micrometer-tracing-bridge-otel
  - opentelemetry-exporter-otlp
config:
  management.tracing.sampling.probability: 0.1
  management.tracing.propagation.produce: w3c
  management.tracing.propagation.consume: w3c,b3
  management.otlp.tracing.endpoint: http://otel-collector:4318/v1/traces

Three knobs: sampling rate, propagation format, export endpoint. Dual propagation (w3c + b3) eases migration.

Sampling Strategy: Tiered by Traffic Type

Core transaction paths : 100% — cannot afford to miss errors

Read/query APIs : 10% — sufficient for latency diagnosis

Health checks/static assets : 0% — pure noise

Load test traffic : 100% — tagged for later filtering

Better than global rate: route-level sampling at gateway via OTel Collector tail-sampling; application stays simple.

Baggage for Business Context

Propagate fields like user-tier, region across services without manual threading. Configure allowed fields; keep payload small (sent on every outbound request).

Custom Spans: Restraint Principle

Create spans only for "nodes where you need to know latency when things break": external calls, slow queries, retry blocks. Method-level full coverage creates unreadable "spiderweb" trace graphs.

Span span = tracer.nextSpan().name("refund.handle").tag("order.no", no).start();
try (var ws = tracer.withSpan(span)) {
  refundService.refund(no);
} catch (Exception e) {
  span.error(e); throw e;
} finally { span.end(); }

Logging Implementation: Layered Levels, Async Safety, Dynamic Debug

Four Log Layers

Access : Request/response params, latency → Structured log platform

Business : State transitions, key decisions → Structured log platform

System : Exceptions, GC, pool saturation → Alerting pipeline

Debug : Intermediate variables → Off by default; enabled per traceId

Mixing layers into one level causes "lots of logs, no signal".

Async MDC Propagation

Wrap thread pools with TaskDecorator that snapshots MDC on submit, restores on execute, clears in finally. Apply globally via AsyncConfigurer for @Async.

ex.setTaskDecorator(runnable -> MdcAware.wrap(runnable));
// MdcAware.wrap: copy MDC map -> set in child thread -> clear after

Dynamic Log Level per Trace

Admin endpoint activates DEBUG for a specific logger + traceId, with auto-revert timer. Workflow: alert → get traceId → identify service → enable DEBUG for that trace → reproduce. Orders of magnitude faster than config change + restart.

Log Sampling Trade-offs

High-volume success logs (one line per request) are noise in steady state. Options: (1) steady-state 1/100 sampling, full on error; (2) move success details to metrics/traces, log only state changes and exceptions. Option 2 is cleaner but requires mature metrics/tracing usage.

End-to-End Example: Order Service Integration

Goal: from alert to code line in <5 minutes. Dependencies: micrometer-registry-prometheus, micrometer-tracing-bridge-otel, opentelemetry-exporter-otlp, logstash-logback-encoder.

Business Instrumentation

@Service
public class OrderService {
  private final ObservationRegistry registry;
  public String create(OrderCmd cmd) {
    return Observation.createNotStarted("order.create", registry)
      .lowCardinalityKeyValue("channel", cmd.channel())
      .highCardinalityKeyValue("order.no", cmd.orderNo())
      .observe(() -> doCreate(cmd));
  }
}

Channel (bounded) goes to metrics; orderNo (unique) goes only to trace.

Structured Log Pattern

<pattern>
{"ts":"%d{yyyy-MM-dd HH:mm:ss.SSS}",
 "level":"%level",
 "traceId":"%X{traceId:-}",
 "spanId":"%X{spanId:-}",
 "svc":"order",
 "logger":"%logger{24}",
 "msg":"%msg"}
</pattern>

svc field mandatory : shared log platform needs service name for cross-service queries. Prefer LoggingEventCompositeJsonEncoder over manual JSON template.

Validated Troubleshooting Path

Alert: order P99 > 800ms (Prometheus + Alertmanager)

Dashboard drill-down: order.create latency spike, tag channel=app Jump to trace: pick slow trace → bottleneck at inventory.check cross-service call, downstream Redis timeout retry ×3

Jump to logs via traceId: see I/O thread pool saturation WARN

Root cause + fix identified — total 4 minutes

Anchors: metric name aligns with span name (dashboard→trace); traceId threads through logs (trace→log). Break either anchor and the path fails.

Eight Common Pitfalls & Diagnostic Playbook

userId in metric labels → cardinality explosion → Prometheus OOM. Fix: labels must be bounded.

Async thread traceId loss → MDC is ThreadLocal. Fix: TaskDecorator / context-propagation.

Reactor stream traceId break → enable hooks.onEachOperator for auto context propagation.

Full sampling → storage cost spiral. Fix: tiered sampling from day one.

Method-level span everywhere → unreadable trace graph. Fix: spans only at key nodes.

Metric name ≠ span name → dashboard-to-trace jump broken. Fix: align naming.

Plain text logs → traceId not queryable → trace-to-log becomes manual grep. Fix: structured JSON with traceId field.

Health checks mixed with business metrics → probe traffic pollutes cardinality & alerts. Fix: separate management.server.port.

Symptom-Based Diagnostics

Trace breaks after service X → verify tracing bridge + propagation format alignment.

Custom metric missing → check Observation name, registry injection, handler registration; use --debug to list autowire candidates.

traceId empty in logs → verify pattern %X{traceId}, filter execution, request path not bypassing filter.

Log cross-contamination in thread pool → ensure TaskDecorator clears MDC in finally.

Debug Switches

management.tracing.enabled=true
logging.level.io.micrometer.tracing=DEBUG
# Inspect registered metrics
GET /actuator/metrics/order.create

Learning Path & Interview Essentials

Four-Stage Mastery

Visible (3 days) : three-piece integration + Grafana panels; verify traceId searchable in logs.

Connected (1 week) : metric→trace→log jump loop closed; async MDC propagation working; 4-minute slow-request diagnosis.

Governed (2 weeks) : cardinality reviews + MeterFilter; tiered sampling + dynamic logs; cost curve stable.

Operational (ongoing) : SLO-based alerting; chaos engineering drills; red/blue team exercises reusable.

12 Core Interview Questions (with answers)

What question does each signal answer? → Health / Which segment / Details; granularity & cost increase together.

What problem does Observation API solve? → Single instrumentation emits both Metric & Span; eliminates 2.x dual annotations.

Why micrometer-tracing replace Sleuth? → Sleuth deprecated; functionality merged; works with Observation unified entry.

OTel vs Brave bridge? → New projects: OTel (protocol standardization); existing Zipkin deep customization: Brave.

traceId vs spanId mutation? → traceId constant end-to-end; spanId new per hop.

Why no userId in metric labels? → Cardinality explosion; series count = product of label value counts.

lowCardinality vs highCardinality? → Former to metrics+trace; latter to trace only.

Why MDC lost? How to fix? → ThreadLocal doesn't cross threads; TaskDecorator / context-propagation.

Head vs tail sampling? → Head decides at ingress; tail decides at backend after seeing result (e.g., keep all errors).

Timer vs Counter? → Need latency distribution → Timer; Counter has no distribution.

Custom span principle? → Only where you need latency on failure; avoid spiderweb.

Good observability landing standard? → Alert to log context in ≤3 jumps, each jump anchored by an ID.

One-Page Cheat Sheet

Essence : Three signals, one request — correlation makes it observable. Three Questions : Overall health / Which segment / Moment details. Entry : Observation unified instrumentation — one instrumentation, two signals. Propagation : traceId constant, spanId per hop — single header. MDC : ThreadLocal, async loses — TaskDecorator safety net. Two Key Tiers : lowCardinality → metrics; highCardinality → trace only. Metric Iron Law : Label values must be finite set. Sampling : Tiered by traffic — core 100%, read 10%, noise 0%. Logs : Structured JSON, traceId queryable, svc field mandatory. Jump Anchors : Metric name aligns span name; traceId threads through logs. New Projects : OTel bridge + OTLP export — backend swappable, app unchanged.

Final takeaway: Observability is not installing three tools; it's ensuring every alert can follow ID anchors through "Metrics → Tracing → Logs" to the last hop. Get that path working, tool choice becomes secondary.