Spring Cloud Tracing: TraceId Propagation Across Services, Threads & MQ

Introduction

Observability relies on three pillars: metrics tell you how much is wrong, logs tell you which line is wrong, but only tracing tells you which service and code segment slowed down a specific request. The core difficulty lies not in collection or visualization but in propagation : a traceId must cross HTTP, thread pools, message queues, and async callbacks; any break splits the trace.

Three Real Incidents

Incident 1: Logs contain traceIds but cannot be stitched. Order, inventory, and payment services each show different traceIds because RestTemplate, OkHttp, and handwritten HttpClient lack a unified header injection entry point.

Incident 2: Async tasks lose traceId. ThreadLocal stores context per thread; switching threads loses the map. Half of all trace breaks stem from this.

Incident 3: 100% sampling fills storage in three days. A 1M DAU app generated billions of spans daily, overwhelming Elasticsearch. Reducing to 0.1% caused missed error traces — probabilistic sampling conflicts with the need to see errors.

Technical Analysis: What Tracing Records

Three Concepts, One Tree

Trace 7f3a9c... (global request id)
├─ Span A gateway 0ms ── 120ms
│ ├─ Span B order 5ms ── 60ms
│ │ └─ Span C db 20ms ── 35ms
│ └─ Span D stock 65ms ─ 110ms
└─ Span E mq-consume 130ms ─ 180ms

Trace: End-to-end request, identified by globally unique traceId.

Span: Work unit with spanId, parentSpanId, start time, duration, tags, events. Spans form a tree via parentSpanId.

SpanContext: Cross-process payload containing only traceId, spanId, sampling flag — no business data. Business data uses Baggage, which inflates headers if abused.

Parent-child relationships are reconstructed via parentSpanId at the backend, not by upload order, making tracing resilient to out-of-order spans and clock skew.

Propagation Formats: W3C vs B3

W3C Trace Context is the standard:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

(version-traceId-spanId-flags). B3 uses multiple headers (X-B3-TraceId, X-B3-SpanId, etc.). Recommendation: use W3C for new systems, enable B3 compatibility via Micrometer's composite propagator; header overhead is worth avoiding broken traces.

Micrometer Observation, Tracing, Bridge Layers

Micrometer Observation unified instrumentation API
ObservationRegistry / ObservationHandler
Micrometer Tracing semantic layer
Tracer / Span / Propagation
Bridge concrete implementation
brave (Zipkin) or otel (OTLP)
Reporter upload
HTTP batch / OTLP gRPC

This layering decouples instrumentation from backend; switching Brave to OTel only changes dependencies. Sleuth is deprecated; migration cost centers on this abstraction.

Solution Matrix

Micrometer Tracing + Brave + Zipkin: Spring official, lowest migration cost, integrates with actuator, Feign, Gateway. Zipkin query limited, storage pressure at scale.

Micrometer Tracing + OTel + OTLP: Standard protocol, broad ecosystem (Jaeger, Tempo, SkyWalking). Suits teams with unified observability platform.

SkyWalking: Java agent zero-code, provides topology, JVM metrics, slow SQL. Couples agent to framework versions; probe overhead higher than SDK.

Selection rule: can modify code & want lightweight → Micrometer; cannot modify code & want full coverage → SkyWalking; need cross-language & standard → OTel. Often combined.

ID Generation & Collision

traceId is 128-bit random (32 hex chars), spanId 64-bit. 128-bit space makes collision probability far lower than disk error rates — no central coordination needed, eliminating single point and bottleneck. Sampling decision must be coordinated: all services in a trace must adopt the same decision via parent-based sampling (flag in propagation header).

Span Count Reality

One order placement easily exceeds 20 spans: gateway 1 + order 1 + 3 RPC + 8 SQL + 6 Redis + 2 MQ. Span count drives storage cost and visual clutter; only instrument operations that may be slow, error-prone, or cross-process.

Core Flow: How a Request's Spans Grow

Entry: Root Span Creation

Server checks for traceparent header. Present → inherit as parent context, create child span. Absent → new trace, become root. Failure usually due to filter order or non-filter protocols.

Exit: Injection at Last Moment

Injection must happen just before request dispatch, via unified interceptors: Feign (ObservationFeignInterceptor), RestTemplate (ClientHttpRequestInterceptor), WebClient (ExchangeFilterFunction), Gateway (GlobalFilter). Handwritten HttpClient requires manual injection — cause of Incident 1.

Span Lifecycle: Start/End Must Pair

Missing end means span never reports (reporters submit on end) and ThreadLocal scope leaks, causing subsequent requests to attach to wrong parent — manifests as scrambled traces, not missing ones.

Reporting: Batched, Async, Discardable

Reporters buffer spans (dozens to hundreds), retry limited times, drop on queue full. Tracing data is intentionally discardable to avoid slowing business. Missing segments may be due to reporter drops — check application logs for discard counters.

Complete Timeline

① Gateway GlobalFilter creates root span
② Forward: inject traceparent into header
③ Downstream Filter extracts: restores parent context
④ Business method creates child spans (DB / Redis / RPC)
⑤ Outbound client injects, continues to next hop
⑥ Each span.end(), handed to reporter for batch upload
⑦ Backend aggregates by traceId, builds tree via parentSpanId

Breakage Map: Five Break Points, Five Symptoms

① Entry no extract → downstream new trace
   Symptom: every call independent trace
② Exit no inject → downstream misses parent context
   Symptom: this service ok, next hop splits
③ Thread switch no restore → child thread traceId null
   Symptom: main flow ok, async segments all null
④ Consumer no extract → async segment new trace
   Symptom: new trace appears after MQ
⑤ Span no end → not reported or parent-child chaos
   Symptom: missing nodes / attached to wrong parent

Map symptoms to break point, locate code in minutes.

Core Source Code: Four Segments Explain Propagation

Tracer: Span Lifecycle

// Entry creates root span
Span span = tracer.nextSpan()
    .name("queryOrder").start();
try (Tracer.SpanInScope ws = tracer.withSpan(span)) {
    // Spans created here become children
    return doQuery(id);
} finally {
    span.end(); // No end = no report
}

withSpan

returns a Scope that pushes span onto ThreadLocal. Try-with-resources pops on close, enabling nested calls (A→B→C, exit C restores B).

ThreadLocalCurrentTraceContext: Why Pairing Matters

public Scope newScope(@Nullable TraceContext ctx) {
    TraceContext prev = currentTraceContext.get();
    currentTraceContext.set(ctx);
    return () -> {
        if (prev != null) {
            currentTraceContext.set(prev);
        } else {
            currentTraceContext.remove();
        }
    };
}

Context is a stack , not a variable: push on enter (save prev), pop on exit (restore prev). Any "set without restore" leaves residue; thread pool reuse then mixes unrelated requests into one trace.

Propagation: Inject & Extract

// Outbound: write current context into header
propagator.inject(
    tracer.currentTraceContext().context(),
    carrier, (c, k, v) -> c.put(k, v));

// Inbound: restore parent context from header
TraceContextOrSamplingFlags extracted
    = propagator.extract(carrier, (c, k) -> c.get(k));
Span span = tracer.nextSpan(extracted);

Carrier abstracts the transport: HTTP headers, message headers, Map — same propagation logic works for HTTP and MQ.

Reporting Chain: SpanHandler Responsibility Chain

After span.end(), spans pass through a chain of SpanHandlers: format conversion (Zipkin), sampling filter, finally reporter queue. Allows inserting handlers for "only error spans" or "filter by service" without changing reporter.

Gateway: Root Span Pitfalls

Order: Tracing filter must run early (negative order) so downstream filters attach to correct parent.

Reactive: Gateway runs on Reactor; span must go into Reactor Context, not ThreadLocal, else forward step loses it.

Retries/LB: Retries create parallel child spans; add attempt tag to avoid misreading as duplicate calls.

Cross-Thread: Five ThreadLocal Break Points

All mechanisms assume current thread's ThreadLocal; thread switching breaks this. Five scenarios from common to subtle:

5.1 Manual Thread Pool

@Bean
TaskDecorator traceTaskDecorator() {
    return runnable -> {
        ContextSnapshot snapshot = ContextSnapshot.captureAll();
        return snapshot.wrap(runnable);
    };
}

TaskDecorator

wraps tasks at submit time: captureAll runs on submitting thread (captures caller's traceId), wrap restores on execution thread. Only works for Spring's ThreadPoolTaskExecutor; raw ThreadPoolExecutor, ForkJoinPool, third-party pools need manual wrapping.

5.2 @Async with Custom Pool

@Bean
ThreadPoolTaskExecutor bizExecutor(TaskDecorator traceTaskDecorator) {
    ThreadPoolTaskExecutor ex = new ThreadPoolTaskExecutor();
    ex.setTaskDecorator(traceTaskDecorator);
    ex.initialize();
    return ex;
}

Must specify executor in @Async("bizExecutor"). Default executors lack decorator, causing null traceId.

5.3 CompletableFuture

Default uses ForkJoinPool.commonPool() — cannot decorate globally. Pass a decorated executor explicitly:

// Default: context lost
CompletableFuture.supplyAsync(this::call);
// Correct: pass wrapped executor
CompletableFuture.supplyAsync(this::call, wrappedExecutor);

5.4 Reactor: Context Not ThreadLocal

Reactive pipelines run on multiple threads; ThreadLocal useless. Reactor uses its own Context, written via contextWrite, read via deferContextual. Micrometer auto-integrates standard operators, but manually created Mono or handwritten subscribe still break.

return webClient.get().retrieve()
    .bodyToMono(Order.class)
    .contextWrite(ctx -> ctx.put(
        TraceContext.class,
        currentTraceContext()));

5.5 Scheduled Tasks & MQ Consumer Threads

Both lack upstream context. Scheduled tasks should start fresh trace per execution (not chain to previous). MQ consumer extracts traceparent from message headers — if present, continue; else new trace. Critical: consumer must clean up scope via try-with-resources, else thread reuse pollutes next message (message B attaches to message A's trace).

5.6 Scheduled Tasks & Raw Threads

@Scheduled(cron = "0 0 2 * * ?")
@Traceable // New trace each run
public void syncJob() { ... }

// Raw thread: must wrap manually
Runnable r = ContextSnapshot.captureAll()
    .wrap(() -> doWork());
new Thread(r).start();

Scheduled tasks = independent traces (otherwise month-long trace with 30 same-name spans). Raw threads = derived work, must inherit context. Same mechanism, opposite intent.

Cross-MQ: Stitching Async Traces

Sync uses HTTP headers; async uses message headers. Same principle, more pitfalls: batching, retries, delays, dead-letter queues.

6.1 Producer Injection

// Kafka: inject traceparent into record headers
ProducerRecord<String, String> rec
    = new ProducerRecord<>(topic, key, value);
propagator.inject(currentContext(),
    rec.headers(), (h, k, v) ->
    h.add(k, v.getBytes(UTF_8)));
kafkaTemplate.send(rec);

Kafka uses RecordHeaders, RocketMQ userProperties, RabbitMQ MessageProperties headers — all key-value, fit traceparent. Spring Cloud Stream / Spring Kafka auto-inject if using their Template/Binder; manual producer loses it.

6.2 Consumer Extraction & Cleanup

TraceContext ctx = extractor.extract(record.headers());
Span span = tracer.nextSpan(ctx)
    .name("consume-order").start();
try (Tracer.SpanInScope ws = tracer.withSpan(span)) {
    handle(record.value());
} finally {
    span.end();
}

Three mandatory actions: extract (restore parent), create consumer span (name indicates consumption), cleanup (close scope to prevent thread-reuse pollution). Missing any yields half-broken trace.

6.3 Batch Consumption Trade-offs

Batch of 100 messages: one span for whole batch (simple, loses per-message latency) vs 100 spans (precise, wide trace). Compromise: parent span + child spans only for timed-out messages. Avoid single span with per-message errors — then you cannot tell which message failed.

6.4 Retries, DLQ, Trace Length

Retries reusing same traceId create extremely long traces, indistinguishable retry count. Better: new span per retry with retry.count tag, child of original. Dead-letter queue → new trace (different processing path).

6.5 Full Cross-Service + MQ Trace Example

gateway (root)
└─ order-service
  ├─ order-db select
  ├─ stock-service (Feign)
  │ └─ stock-db update
  └─ kafka-send order.created
  └─ pay-consume (async continuation)
    └─ pay-db insert

Note: pay-consume is child of kafka-send, not order-service — shows "consumption triggered by this send", possibly seconds/hours later. Trace captures cross-time causality, unlike logs.

Sampling: Balancing Cost & Visibility

7.1 Head vs Tail Sampling

Head sampling: Decide at trace start, write decision to propagation header, all downstream inherit. Constant overhead, simple; but decision made before knowing if trace will error — errors likely dropped.

Tail sampling: Collect all spans first (in Collector/Agent), decide after trace completes. Can keep all errors, all slow requests, sample normal at 1%. Requires buffering component (OTel Collector tail sampling processor), extra memory and latency.

Production hybrid: gateway head-samples (control volume), critical services force-sample, collector tail-samples with "error must keep" rule.

7.2 Sampling Rate Math

Daily spans = QPS × 86400 × spans-per-trace × sample-rate
Example: 200 QPS × 86400 × 30 × 0.1 ≈ 13.8M spans/day
~1KB per span → ~14 GB/day

Reducing rate from 1.0 to 0.1 may only halve storage because error/slow traces have more spans (retries, fallbacks) and dominate volume.

7.3 Sampling Decision Only at Entry

If each service rolls dice independently, 8-layer trace full capture probability = 0.1^8 ≈ 0. Decision made once at entry, written to traceparent flags (01 = sampled). Downstream inherits unconditionally — parent-based sampling.

management:
  tracing:
    sampling:
      probability: 0.1 # Only for new traces
    zipkin:
      tracing:
        endpoint: ${ZIPKIN_ENDPOINT}

Config probability applies only when creating new trace; existing parent context overrides it — often mistaken as "config not working".

7.4 Three Ways to Guarantee Error Sampling

Collector tail sampling: OTel Collector tail_sampling processor with rules: keep errors, keep high latency, sample rest at 1%. Most complete but needs buffering and consistent routing across replicas.

Agent-side: SkyWalking agent supports forced sampling for errors/slow requests, zero code change. Strategy opaque, tuning less intuitive.

Application-side active marking: Catch exception, mark span error, call force-sample API. Flexible but requires code at every error point; easy to miss.

If budget allows only one thing: do "error must sample". Normal traces rarely reviewed; error traces used daily. Allocate sampling budget to errors and slow requests.

Eight Common Pitfalls

Assuming dependency alone enables tracing — need filters, interceptors, unified HTTP client.

Manually created thread pools don't inherit context — must wrap. @Async without explicit executor falls to undecorated default.

Forgetting span.end() — span unreported, scope leaks, scrambles subsequent traces.

Thread pool reuse leaves traceId residue — next task inherits wrong trace.

MQ consumer doesn't extract message headers — async segment forever broken.

Sampling rate 1.0 — storage explodes, CPU/network overhead forces rollback.

Tracing without log correlation — traceId not in log pattern, can't jump from slow trace to logs.

Troubleshooting Three Steps

Add %X{traceId} to log pattern, verify per-machine context.

Enable logging.level.io.micrometer.tracing=DEBUG to see inject/extract calls and results.

Capture raw request headers, check if traceparent dropped by gateway, proxy, or middleware.

Follow "entry extract → business execute → exit inject" to locate in minutes.

Learning Path & Interview Questions

Four-Stage Roadmap

Beginner (3 days): Integrate Brave+Zipkin, connect gateway to three services, put traceId in log pattern.

Intermediate (1 week): Cover four break points: custom thread pool, @Async, CompletableFuture, Reactor; connect Kafka producer/consumer.

Advanced (2 weeks): Read four source sections: Tracer lifecycle, ThreadLocalCurrentTraceContext stack restore, Propagation inject/extract, SpanHandler chain; understand W3C vs B3 differences.

Practice (ongoing): Design sampling strategy (head + tail + error keep), calculate storage cost; run "intentional break" drill to verify monitoring detects it.

Twelve Interview Questions (with Answers)

Key Q&A covered: Trace/Span relationship, traceId generation/collision, cross-service propagation via headers, async traceId loss due to ThreadLocal, thread pool context propagation via TaskDecorator, @Async executor requirement, Reactor Context vs ThreadLocal, MQ trace stitching (producer inject, consumer extract + cleanup), consumer cleanup necessity, head vs tail sampling, parent-based sampling rationale, troubleshooting steps.

One-Page Cheat Sheet

Model: Trace a tree, Span a node
  Parent-child via parentSpanId
Propagation: W3C traceparent four parts
  version-traceId-spanId-sample-flag
Entry extract, exit inject, carrier swappable
Context is stack: enter set, exit restore prev
  Only set no restore -> thread pool pollution
Cross-thread five breaks: custom pool / @Async
  / CompletableFuture / Reactor / MQ
Unified fix: capture at submit, restore at exec
  Cleanup after, try-with-resources
Reactor uses Context, not ThreadLocal
MQ: producer inject header, consumer extract + cleanup
Sampling: entry decides once, downstream inherits
  Pure head misses errors, add tail sampling

Summary

Tracing's difficulty is not "add a dependency" but "keep context alive across every thread switch and protocol hop". Remember three rules: context is a stack not a variable, cross-thread requires capture-restore, sampling decision only at entry. All configurations and pitfalls are variants of these three.