Skip to content
Distributed Tracing in Production: SkyWalking vs Jaeger Implementation Deep Dive
Operations 60 min read

Distributed Tracing in Production: SkyWalking vs Jaeger Implementation Deep Dive

This comprehensive guide covers distributed tracing implementation with SkyWalking and Jaeger, detailing trace data costs, head vs tail sampling trade-offs, architecture differences, capacity planning formulas, probe integration pitfalls, cross-language trace breakage causes, and three real-world troubleshooting cases with concrete metrics and configurations.

Ops Community
Ops Community
Ops Community

Part 1: The Physical Essence of Traces — Why Full Instrumentation Fails

1.1 Byte Cost of a Single Trace Determines Architectural Limits

The primary engineering contradiction in distributed tracing is not functionality but the severe asymmetry between data volume and value density. A complete trace consists of multiple spans; each span's minimum attribute set includes traceId, spanId, parentSpanId, serviceName, operationName, startTime, duration, status, several tags, and possible logs (events).

Single Span minimum serialization overhead (OTLP protobuf estimate)
  traceId       16 B
  spanId         8 B
  parentId       8 B
  name+service  ~40 B
  timestamps    16 B
  status         4 B
  basic tags    ~80 B
  ------------------------
  Total        ~172 B (without business attributes)

A cross-service call chain spanning 6 services: 8-12 spans
  -> Each trace ~1.6 KB ~ 2.1 KB
  At 3000 QPS, full collection:
  -> 3000 x 2 KB x 86400 = 518 GB/day (uncompressed)

Write amplification: Jaeger writing to Elasticsearch stores each span as a document; index and _source each store a copy, actual storage ~1.8-2.5x raw data. 518 GB raw becomes 1 TB+ on disk . This is why "collect all first, optimize later" fails by week two due to disk alerts.

1.2 Sampling: Mechanistic Divergence Between Head and Tail Sampling

Head-based sampling (SDK-side decision)
  client -> SDK rolls dice (rate=0.01)
    -> decides traceId last 8 bits, all services in chain
       inherit same sampling decision via traceId
  Pros: zero extra transport, controllable cost
  Cons: decision made without knowing outcome; slow requests not retained "after the fact"

Tail-based sampling (Collector-side decision)
  client -> full report -> Collector caches complete trace in memory/disk
    -> judges (duration > 500ms or status=ERROR)
    -> only retains matching traces, discards rest
  Pros: 100% retention of slow and error traces
  Cons: Collector needs caching and stateful routing

Tail sampling is the current production standard but its cost is often underestimated: Collector must guarantee all spans of the same traceId land on the same instance (otherwise judgment sees only half a chain), so a loadbalancing exporter performing traceId consistent hashing is required upfront.

skywalking-oap x2 <-(gRPC 11800)- agent
    |
    +-- BanyanDB / ES storage layer (sharded)

OTel Collector (gateway layer, hash by traceId)
  receiver: otlp
  processors: [memory_limiter, batch, tail_sampling]
  exporters: [loadbalancing(traceID) -> collector-pool]
      |
      collector-pool x3 (stateful, cache complete trace)
      |
      exporter: jaeger-storage / otlp -> ClickHouse

1.3 Architectural Foundation Differences: SkyWalking vs Jaeger

SkyWalking 10 (Apache Top-Level Project)
  Agent (Java/PHP/Go/nginx-lua/eBPF Rover)
    | gRPC 11800 / HTTP 12800
  OAP (Streaming analytics engine: produces metrics directly)
    | Storage abstraction layer
  Storage: BanyanDB (custom) | ES (mainstream) | MySQL (small scale)
    | GraphQL 12800
  UI: 8080

Jaeger v2 (Built on OTel Collector)
  Agent/SDK (OTel SDK, official client libs archived)
    | OTLP 4317(gRPC) / 4318(HTTP)
  jaeger-v2 (Config-driven single binary)
    | Storage plugins
  Storage: ES | OpenSearch | Cassandra | ClickHouse | Badger
    | Query 16686
  UI: 16686

Two decisive architectural differences:

SkyWalking includes streaming aggregation : OAP computes metrics (service SLA, topology, endpoint metrics) while receiving spans, making it a native APM without extra components.

Jaeger v2 is "Collector + storage plugins" : It only handles receive, process, write; metric aggregation relies on Prometheus (via spanmetrics connector), topology view weaker than SkyWalking.

1.4 2026 Version Prerequisites (Must Align First)

Key version statuses:

Jaeger v1 : EOL 2025-12-31 -> must migrate to v2, config changes from CLI args to YAML.

Jaeger v2 : Based on OTel Collector -> reuses entire Collector component ecosystem.

SkyWalking 10.x : Current mainline; H2 storage removed, minimum requires ES.

SkyWalking BanyanDB : GA; official recommended custom storage, eliminates ES ops.

OTel Java Agent 2.x : Jaeger official client libraries archived. H2 storage removed is critical: old tutorials using storage: h2 fail to start on 10.x; must switch to BanyanDB or Elasticsearch.

1.5 Three Iron Laws of Implementation

Sampling rate must be a calculable number , not "set 100% first". Calculation method in Chapter 3.

Spans must have unified semantic conventions , otherwise cross-language traces break ( parentSpanId loss is the most common cause of broken chains).

Storage must be rotatable ; traces are the only observability data that must expire and be deleted ; retention period directly determines capacity.

Part 2: Production Baseline — Three-Layer Topology, Versions & Capacity Assumptions

2.1 Environment Assumptions

Business scale: 42 microservices, 31 Java services (Spring Boot 3.2)

Peak QPS: total 3200, single service max 900

Nodes: K8s v1.31, 8 nodes, each 16C64G

Storage: Elasticsearch 8.15 cluster (3 hot + 2 warm)

SkyWalking: OAP 10.2, BanyanDB as backup

Jaeger: v2.1 (OTel Collector distribution)

Reporting protocol: OTLP gRPC 4317 (unified entry)

2.2 Topology & Ports

[Business Pod] --OTLP:4317--> [otel-gateway svc] --> [jaeger-v2]
    |                                    (tail_sampling)      :16686 UI
    +--gRPC:11800----> [skywalking-oap svc] :12800
                          |                    (GraphQL)
                          v
                    [banyandb / es]
                          ^
                          |
                    [skywalking-ui :8080]

Required ports (often missed in K8s NetworkPolicy):

4317 gRPC - OTLP trace reporting

4318 HTTP - OTLP trace HTTP reporting

11800 gRPC - SkyWalking Agent reporting

12800 HTTP - OAP GraphQL query

17912 gRPC - BanyanDB data node

16686 HTTP - Jaeger Query UI

2.3 OS & Kernel Baseline: Collector is "Memory-Sensitive"

Tail sampling turns Collector from stateless to stateful; memory and connection counts become primary bottlenecks. Baselines for 16C64G Collector nodes:

# /etc/sysctl.d/90-otel.conf
# Apply: sysctl --system

# --- gRPC long connections dense, port range & TIME_WAIT pressure ---
net.ipv4.ip_local_port_range = 10240 65000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.core.somaxconn = 8192
net.ipv4.tcp_max_syn_backlog = 8192

# --- Collector memory jitter large, prevent premature OOM ---
vm.overcommit_memory = 1
vm.swappiness = 1
vm.max_map_count = 262144

# --- Heavy UDP for DNS resolution (service discovery) ---
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
# K8s: Collector Deployment resource config (must explicitly limit)
resources:
  requests:
    cpu: "4"
    memory: 8Gi
  limits:
    # Note: limits and memory_limiter hard limit must leave gap
    memory: 12Gi
resources.limits.memory

and Collector's memory_limiter must be staggered: memory_limiter.limit set to 75% of limit, otherwise cgroup OOMKill happens before graceful data drop.

2.4 Four-Dimensional Breakdown: SkyWalking OAP Config

# config/application.yml (SkyWalking 10.2, key snippets)
# Effective: OAP restart (no hot reload)

core:
  default:
    # [Must change] default 128, must increase when services >200
    restHost: 0.0.0.0
    restPort: 12800
    gRPCHost: 0.0.0.0
    gRPCPort: 11800
    # [Tuning] Data TTL, calculate by retention & disk
    recordDataTTL: 3
    metricsDataTTL: 7

storage:
  selector: ${SW_STORAGE:banyandb}
  banyandb:
    targets: ${SW_STORAGE_BANYANDB_TARGETS:banyandb:17912}
    # [Tuning] Batch write size, too large causes single batch timeout
    flushInterval: 3000

receiver-trace:
  default:
    # [Must change] Even after sampling, may be huge; must rate-limit
    sampleRate: ${SW_TRACE_SAMPLE_RATE:5000}
    slowDBAccessThreshold:
      ${SW_SLOW_DB_THRESHOLD:default:200,mongodb:100}

agent-analyzer:
  default:
    # [Tuning] Slow trace threshold, align with business SLA
    traceAnalysis:
      segmentThreshold: ${SW_SEGMENT_THRESHOLD:500}

2.5 Four-Dimensional Breakdown Table

① Must-Change Items storage.selector: banyandb or elasticsearch — default H2 removed in 10.x sampleRate: calculate via formula — default value blows queue at peak recordDataTTL: 3 days — default 3 days but often mistakenly increased restHost: explicit listen address — default 127.0.0.1 makes UI unreachable

② Coupled Items sampleRate with receiver-trace queue length ( bufferChannelSize): lower sampling rate requires smaller queue, else memory wasted. recordDataTTL with disk capacity: 3 days x daily write x storage amplification = minimum disk (see Ch3 formula). metricsDataTTL with dashboard query window: TTL < window causes "curve truncation". slowDBAccessThreshold with DBA slow query threshold: inconsistency leads to same SQL judged differently in APM vs DB.

③ Performance Tuning Items (Formulas)

Sampling rate = min(target daily data / full daily data, 1)

Target daily data = tolerable disk growth / retention days / amplification factor
Baseline: 2TB available, retain 3 days, amplification 2.0
  -> Target daily = 2048GB / 3 / 2.0 = 341 GB/day
Full daily data = QPS x avg trace size x 86400
  -> 3200 x 1.8KB x 86400 = 497 GB/day
  -> Sampling rate = 341 / 497 = 0.686

Actual take 0.5 (leave 30% margin), i.e., sampleRate=5000
(SkyWalking unit is 1/10000)

④ Absolute Red Lines (Never Change) storage.selector switch mid-way -> historical data unqueryable, requires rebuild. core.default.role (cluster mode) wrong -> OAP split-brain. recordDataTTL set to 0 -> data expires immediately and gets deleted. traceId generation logic altered -> trace breaks, cannot correlate.

Part 3: Capacity, Sampling & Retention — Three Numbers Must Close the Loop

3.1 Disk Estimation Formula (Must Calculate Before Landing)

Required disk = daily write x retention days x storage amplification x (1 + 25%)

Storage amplification:
  Elasticsearch 1.8 ~ 2.5 (1 replica: 3.6 ~ 5.0)
  ClickHouse     1.0 ~ 1.3
  BanyanDB       0.6 ~ 1.0 (columnar compression, best)

Baseline calc (ES, 1 replica):
  341 GB/day x 3 days x 4.0 x 1.25 = 5115 GB ≈ 5 TB

This result forces storage re-evaluation. BanyanDB compression advantage shines: same data needs only ~1.2 TB.

Storage comparison:

BanyanDB: compression 0.6-1.0, low ops (no JVM tuning), preferred for medium-large scale.

ClickHouse: 1.0-1.3, medium ops (needs expertise), large scale & analytical.

Elasticsearch: 1.8-2.5, high ops (index tuning), reuse existing ES cluster.

Cassandra: 2.0-3.0, medium-high ops, legacy architecture, not recommended for new builds.

3.2 Retention & Query Performance Coupling

recordDataTTL = 3

means querying "7-day-old traces" always fails, but postmortems often need 7-day data. Solution is not extending TTL but tiered retention :

# Short-term: full attributes, 3 days (ES hot nodes)
# Mid-term: only ERROR & P99 slow traces, 30 days (warm nodes)
# Long-term: only metrics aggregates, 1 year (monitoring DB)

Implemented via two-level exporters in Collector:

processors:
  filter/keep_errors_and_slow:
    error_mode: ignore
    traces:
      span:
        - 'status.code != STATUS_CODE_ERROR and duration < 500ms'
exporters:
  otlp/short_term:
    endpoint: jaeger-hot:4317
  otlp/medium_term:
    endpoint: jaeger-warm:4317

3.3 Sampling Rate Verification: Don't Trust Estimates, Instrument

Check actual receive rate via SkyWalking OAP GraphQL:

curl -s -X POST http://172.16.40.11:12800/graphql \
  -H 'Content-Type: application/json' \
  -d '{"query":"{getClusterBrief(
    duration:{start:\"2026-11-18 00\",end:\"2026-11-18 01\"})
    {numOfServices}}"}' | jq .

Jaeger v2 Collector internal metrics (must enable telemetry):

curl -s http://172.16.40.21:8888/metrics | \
  grep -E 'otelcol_(receiver_accepted|processor_)'
# Output example:
otelcol_receiver_accepted_spans{receiver="otlp"} 18241300
otelcol_processor_tail_sampling_count_spans{decision="sampled"} 9121
otelcol_processor_tail_sampling_count_spans{decision="dropped"} 18232179
sampled / (sampled + dropped)

= actual retention rate. If deviation from configured 5% exceeds 50%, tail_sampling policy likely wrong (usually decision_wait too short causing incomplete traces).

Verify

decision_wait</sub> adequacy (trace completeness):</p><pre><code>curl -s http://172.16.40.21:8888/metrics | \
  grep 'otelcol_processor_tail_sampling_sampling_trace_dropped'
# Output:
otelcol_processor_tail_sampling_sampling_trace_dropped_too_early 1841

too_early count growing means traces not fully collected before decision_wait expires; increase from 30s to 60s.

3.4 Complete Collector Tail Sampling Config

# /etc/otel/config.yaml (Jaeger v2 / OTel Collector common)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 6144  # 75% of limits
    spike_limit_mib: 1024
  batch:
    timeout: 1s
    send_batch_size: 4096
    send_batch_max_size: 8192
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    expected_new_traces_per_sec: 3000
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 500}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      static:
        hostnames:
          - jaeger-pool-0:4317
          - jaeger-pool-1:4317
          - jaeger-pool-2:4317

service:
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [loadbalancing]

Note: expected_new_traces_per_sec must match actual; used for hash table pre-allocation; too small causes frequent resizing and GC jitter.

Part 4: Probe Integration — From Three Lines of Code to Production Pitfalls

4.1 Java Services: OTel Agent vs SkyWalking Agent Coexistence

Two agents mounted simultaneously interfere (both modify bytecode, both inject context). Production must choose one , or use SkyWalking's OTLP receiving capability as unified entry. Option A: Only SkyWalking Agent (OAP native, most complete topology):

java -javaagent:/opt/skywalking-agent/skywalking-agent.jar \
  -Dskywalking.agent.service_name=order-svc \
  -Dskywalking.collector.backend_service=172.16.40.11:11800 \
  -jar /app/order-svc.jar

Option B: Only OTel Agent, report via OTLP to SkyWalking OAP (SkyWalking 10 supports OTLP receiver, but topology weaker than native):

java -javaagent:/opt/otel/opentelemetry-javaagent.jar \
  -Dotel.service.name=order-svc \
  -Dotel.exporter.otlp.endpoint=http://172.16.40.21:4317 \
  -Dotel.exporter.otlp.protocol=grpc \
  -Dotel.traces.sampler=parentbased_traceidratio \
  -Dotel.traces.sampler.arg=0.05 \
  -jar /app/order-svc.jar

K8s injection method:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-svc
  namespace: app-prod
spec:
  replicas: 6
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        resource.opentelemetry.io/service.version: "1.42.0"
    spec:
      containers:
        - name: order-svc
          image: harbor.example/app/order-svc:1.42.0
          env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://otel-gateway.observability:4317"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "deployment.environment=prod"

Verify injection without pod restart:

kubectl -n app-prod exec deploy/order-svc -- \
  sh -c 'tr "\0" "
" < /proc/1/cmdline | grep javaagent'
# Output:
-javaagent:/otel-auto-instrumentation-java/javaagent.jar

4.2 Nginx: Entry Layer Trace Start Most Easily Lost

Nginx doesn't generate traceId, only propagates and acts as upstream span. Use ngx_http_opentelemetry_module or skywalking-nginx-lua to generate entry span.

# /etc/nginx/conf.d/app.conf (key snippet)
# Effective: nginx -s reload (no traffic break)

# Generate or propagate W3C traceparent
map $http_traceparent $otel_traceparent {
  default $http_traceparent;
  "" "-";
}

server {
  listen 443 ssl;
  server_name app.example.com;

  location /api/ {
    # If no upstream traceparent, module generates
    opentelemetry_operation_name "nginx-proxy";
    opentelemetry_span_attributes
      "http.request.method=$request_method";

    proxy_set_header traceparent $otel_traceparent;
    proxy_set_header X-Request-Id $request_id;
    proxy_pass http://order-svc.app-prod:8080;
  }
}

Test and reload:

nginx -t && nginx -s reload
# Output: nginx: /etc/nginx/nginx.conf test is successful

Verify entry span generation:

curl -s -o /dev/null \
  -H 'traceparent: 00-4bf92f3577b34da6-00f067aa0ba902b7-01' \
  https://app.example.com/api/orders

4.3 Four Typical Causes of Cross-Language Trace Breakage

Context not propagated : downstream becomes new root span -> check if parentSpanId is empty.

Middleware unsupported : MQ/thread pool breaks chain -> check if header carries traceparent.

Async not wrapped : Future breaks chain -> check sudden span count drop.

Sampling decision inconsistent : partial spans missing -> compare sampling rate configs across services.

Most common pitfall: thread pools . Java @Async or CompletableFuture.supplyAsync don't propagate Context by default; must wrap in decorator:

// Thread pool context propagation (production must-do)
Executor wrapped = Context.taskWrapping(
    Executors.newFixedThreadPool(32));

Quick verification of broken traces in a service: count span distribution per traceId:

kubectl -n observability exec deploy/jaeger-v2 -- \
  wget -qO- 'http://localhost:16686/api/traces?service=order-svc&limit=20' \
  | jq '[.data[].spans | length] | group_by(.) | map({spans: .[0], n: length})'
# Output:
[
  {"spans": 3, "n": 14},
  {"spans": 8, "n": 5},
  {"spans": 1, "n": 1}  <-- [Key anomaly] only 1 span indicates breakage
]

4.4 Sidecar vs Gateway Trade-offs

Sidecar : good isolation, independent rate limiting; but per-pod memory overhead; suits high-traffic single service.

DaemonSet : node-level aggregation, resource efficient; but node failure affects whole slice; suits medium scale general.

Gateway : centralized tail sampling; but single-point pressure; suits need for full-view analysis.

Production practice: DaemonSet for batch processing + Gateway for tail sampling . DaemonSet only does batch and memory_limiter ; Gateway does tail_sampling , avoiding each node caching complete traces.

Part 5: High-Frequency Commands & Monitoring Instrumentation

5.1 Service Topology & Dependencies: What to Look at First

SkyWalking: query topology via GraphQL (better for scripting than UI):

curl -s -X POST http://172.16.40.11:12800/graphql \
  -H 'Content-Type: application/json' \
  -d '{"query":"query{getGlobalTopology(
    duration:{start:\"2026-11-18 0900\",end:\"2026-11-18 1000\"})
    {nodes{id name} calls{source target}}}"}' \
  | jq '.data.getGlobalTopology.calls[0:3]'
# Output:
[
  {"source": "bmdDU9ydGhlcjE=", "target": "b3JkZXItc3Zj"},
  {"source": "b3JkZXItc3Zj", "target": "aW52ZW50b3J5"}
]

SkyWalking node IDs are base64 encoded; scripts must decode for alert mapping. Jaeger: query recent slow traces by service:

curl -s "http://172.16.40.21:16686/api/traces?service=order-svc&lookback=1h&limit=20&minDuration=500ms" \
  | jq -r '.data[] | [.traceID, (.spans|length), ((.spans | map(.duration)|max)/1000|floor)] | @tsv'
# Output:
4bf92f3577b34da6a3ce  12  1842  <-- 12 spans, max 1842ms
a1c2e3f4506071829304  6   920   <-- [Interpretation] few spans but slow
d91b2c3a4e5f60718293  9   742

5.2 Key Metrics: Must-Have Dashboard Indicators

# 1) Service-level P99 latency (SkyWalking exported to Prometheus)
histogram_quantile(0.99,
  sum by (le, service) (
    rate(skywalking_service_latency_bucket[5m]))) > 0.5

# 2) Collector dropped span rate (>0 means rate limiting triggered)
rate(otelcol_processor_dropped_spans_total[5m]) > 0

# 3) Tail sampling retention rate (should match configured value)
sum(rate(otelcol_processor_tail_sampling_count_spans_total
  {decision="sampled"}[5m]))
/ sum(rate(
  otelcol_processor_tail_sampling_count_spans_total[5m]))

# 4) Collector queue usage (>80% imminent data loss)
otelcol_exporter_queue_size / otelcol_exporter_queue_capacity > 0.8
# 5) Collection endpoint send failure rate
rate(otelcol_exporter_send_failed_spans_total[5m])
  / rate(otelcol_exporter_sent_spans_total[5m]) > 0.01

# 6) OAP write latency (exposed by SkyWalking 10)
skywalking_oap_persistence_timer_bulk_execute_latency_seconds > 5

# 7) Trace incompleteness rate (broken chain monitoring, custom)
sum(rate(trace_span_single_root_total[10m]))
  / sum(rate(trace_span_total[10m])) > 0.05

Last metric trace_span_single_root_total is not official; must produce via Collector's count connector or custom processor. It is the only leading indicator for broken chains , worth investing.

5.3 One Command to Locate "Which Downstream Slowed the Whole Chain"

curl -s "http://172.16.40.21:16686/api/traces/${TID}" \
  | jq -r '.data[0].spans[] | select(.duration > 100000) |
    [.operationName, (.duration/1000|floor), .process.serviceName] | @tsv' | sort -k2 -nr

Output example:

/order/create           1842  order-svc
/order/query-status     1610  order-svc
/db/query order_item    1580  <-- [Key] occupies 86% of parent span
redis.get               12    order-svc

Interpretation logic: Child span exceeds 80% of parent span and has no deeper children -> bottleneck is in that span's own latency (network or downstream processing), not deeper levels. Confirm which SQL (span tags often have db.statement ):

curl -s "http://172.16.40.21:16686/api/traces/${TID}" \
  | jq -r '.data[0].spans[] | select(.operationName|test("db")) |
    .tags[] | select(.key=="db.statement") | .value'
# Output:
SELECT * FROM order_item WHERE order_id = ? AND status IN (...)

5.4 Log & Trace Correlation: Skip This, You're Half-Blind

Trace tells "where slow", logs tell "why slow". Must correlate via traceId .

<pattern>
  %d{HH:mm:ss.SSS} [%thread] %-5level
  [%X{traceId:-none}] %logger{36} - %msg%n
</pattern>

Log output:

09:41:22.118 [http-nio-8080-exec-3] INFO
  [4bf92f3577b34da6a3ce929d0e0e4736] c.e.OrderController
  - order create start

Query all logs across services by traceId in Loki:

TID=$(curl -s "http://172.16.40.21:16686/api/traces?service=order-svc&limit=1" | jq -r '.data[0].traceID')
curl -sG http://172.16.40.31:3100/loki/api/v1/query \
  --data-urlencode "query={namespace=\"app-prod\"} |= \"${TID}\"" \
  | jq -r '.data.result[].values[][1]' | head -20
# Output:
09:41:22 INFO  order create start trace=4bf92f...
09:41:22 WARN  inventory lock wait 402ms trace=4bf92f...
09:41:23 INFO  order create end cost=1842ms

inventory lock wait 402ms is info trace never sees — that's the value of log correlation.

5.5 Storage Layer Health Checks

ES indices & shards (trace indices prone to shard explosion):

curl -s 'http://172.16.40.41:9200/_cat/indices/sw_*?v' | head -8
# Output:
health status index              pri rep docs.count store.size
green  open   sw_segment-20261118  3   1   38410233   41.2gb
green  open   sw_metrics-20261118  2   1   12840912   12.8gb
yellow open   sw_segment-20261117  3   1   41203918   44.9gb
                         ^ [Key anomaly] yellow means replica unassigned

ClickHouse (Jaeger alt storage): check if writes stalled by merges:

kubectl -n observability exec -it clickhouse-0 -- \
  clickhouse-client -q "
SELECT formatReadableSize(sum(bytes_on_disk)) AS disk,
       count() AS parts
FROM system.parts WHERE active AND table='jaeger_spans';"
# Output:
41.2 GiB  1841  <-- [Key anomaly] parts > 300 means merge can't keep up

ClickHouse active parts > 300 causes query latency spike; increase async_insert or reduce insert batch frequency.

Part 6: Three Deep-Water Troubleshooting Closed Loops

6.1 Case 1: Random Trace Breakage — Root Cause: Async Thread Pool Losing Context

Symptom : order-svc topology shows order-svc -> inventory call volume only 62% of actual business; 38% traces appear as two independent traces. Business reports "occasionally" can't find complete trace. Monitoring Evidence :

# Single-span trace ratio abnormally high
sum(rate(trace_span_single_root_total{service="inventory"}[10m]))
  / sum(rate(trace_span_total{service="inventory"}[10m]))
# Rose from 0.02 to 0.38

Scene Evidence :

# Pull 20 inventory traces, examine span composition
curl -s "http://172.16.40.21:16686/api/traces?service=inventory&limit=20" | \
  jq -r '.data[] | [.traceID, (.spans|length), (.spans[0].operationName)] | @tsv'
# Output:
a1b2c3d4e5f60718293a  1  POST /inventory/lock  <-- [Key] single span
b2c3d4e5f60718293a1b  1  POST /inventory/lock  <-- [Key] single span
c3d4e5f60718293a1b2c  6  /order/create         <-- normal chain

Check inventory received headers (traceparent present?):

kubectl -n app-prod logs deploy/inventory-svc --tail=50 | grep -i 'traceparent\|trace_id'
# Output:
INFO  received request trace_id=  (empty)  <-- [Key] empty
INFO  received request trace_id=4bf92f3577b34da6a3ce929d0e0e4736

Check order-svc call path to inventory:

kubectl -n app-prod logs deploy/order-svc --tail=200 | grep -iE 'async|thread|inventory lock' | head -5
# Output:
INFO [http-nio-8080-exec-7] async submit inventory lock task
INFO [pool-3-thread-2]      inventory lock start
                          ^ [Key] thread pool thread, context lost

Root Cause Deduction : order-svc uses @Async to call inventory asynchronously; Spring's default SimpleAsyncTaskExecutor doesn't propagate Context (i.e., traceparent ), so downstream missing parent context creates a new root span , splitting trace in two. 38% breakage matches async order proportion. Fix :

// 1) Wrap thread pool with Context.taskWrapping
@Bean("inventoryExecutor")
public Executor inventoryExecutor() {
  ThreadPoolTaskExecutor ex = new ThreadPoolTaskExecutor();
  ex.setCorePoolSize(32);
  ex.setMaxPoolSize(64);
  ex.setQueueCapacity(2000);
  ex.setTaskDecorator(runnable -> Context.current().wrap(runnable));
  ex.initialize();
  return ex;
}
# 2) Canary deploy 1 pod, verify breakage ratio
kubectl -n app-prod rollout restart deploy/order-svc
kubectl -n app-prod rollout status deploy/order-svc
# 3) Compare metrics
sum(rate(trace_span_single_root_total{service="inventory"}[10m]))
  / sum(rate(trace_span_total{service="inventory"}[10m]))
# Before fix: 0.38
# After fix: 0.004

Before/After Comparison :

Inventory single-span ratio: 38% -> 0.4%

Topology edge call coverage: 62% -> 99.6%

Avg spans per trace: 5.2 -> 9.4

Rollback : kubectl -n app-prod rollout undo deploy/order-svc . Change only affects context propagation; rollback introduces no data issues.

6.2 Case 2: OAP Queue Full, Massive Trace Drops

Symptom : Peak (09:00-10:00) SkyWalking UI shows topology gaps, segment data missing. Business reports traces completely unqueryable for that period; OAP logs show massive buffer is full . Scene Evidence :

kubectl -n observability logs deploy/skywalking-oap | grep -iE 'queue|reject|overflow|slow' | head -10
# Output:
WARN  DataCarrier  [DataCarrier-ConsumeDriver] buffer is full, dropped 18432 rows
WARN  TraceAnalyzer  segment buffer full, dropped 9211
ERROR GRPCStreamService  status:RESOURCE_EXHAUSTED message: buffer channel size is full

Check OAP JVM & heap:

kubectl -n observability exec deploy/skywalking-oap -- sh -c 'jcmd 1 GC.heap_info | head -6'
# Output:
 PSYoungGen   total 2048000K, used 2011136K
  eden space 1900544K, 99% used  <-- [Key] young gen pressured
 PSOldGen     total 4096000K, used 4012032K

Confirm frequent Full GC:

kubectl -n observability exec deploy/skywalking-oap -- sh -c 'jstat -gcutil 1 1000 5'
# Output:
  S0    S1    E     O     M    CCS   YGC  YGCT  FGC  FGCT
  0.00 98.10 99.32 97.94 93.12 89.44 1842 62.1  41  38.2
                          ^ [Key] Old gen 97%, FGC 41 times

Root Cause : sampleRate kept default (10000 = full), receiver-trace bufferChannelSize default 20000. Peak 3200 QPS x avg 9 spans = 28800 spans/sec, production rate exceeds queue consumption, queue piles up triggering RESOURCE_EXHAUSTED , gRPC side rejects directly, data lost. Fix :

# 1) Lower sampling rate first, emergency stop (config + restart OAP)
kubectl -n observability set env deploy/skywalking-oap SW_TRACE_SAMPLE_RATE=3000
kubectl -n observability rollout status deploy/skywalking-oap
# 2) Increase queue & heap, combined with lower sampling
kubectl -n observability set env deploy/skywalking-oap SW_RECEIVER_BUFFER_CHANNEL_SIZE=60000
kubectl -n observability patch deploy skywalking-oap --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/limits/memory","value":"24Gi"}]'
# 3) Verify no more drops
kubectl -n observability logs deploy/skywalking-oap | grep -c 'buffer is full'
# Output: 0
# 4) Confirm write latency recovered
curl -s http://172.16.40.11:17128/metrics | grep persistence_timer_bulk_execute_latency_seconds | head -3
# Output:
skywalking_oap_persistence_timer_bulk_execute_latency_seconds {quantile="0.99"} 1.82  <-- [Normal] from 42.1 down to 1.82

Before/After :

Queue drops/min: 18432 -> 0

OAP write latency P99: 42.1s -> 1.82s

Full GC (10min): 41 -> 3

Trace retention: ~61% -> 100%

Rollback : kubectl -n observability set env deploy/skywalking-oap SW_TRACE_SAMPLE_RATE- clear variable to restore default. Note: rollback also requires reverting bufferChannelSize , else returns to failure state.

6.3 Case 3: Jaeger Writes ES 429, Traces Only Have First Half

Symptom : 30% of traces in Jaeger UI only have first 3-4 spans, latter half missing; Collector logs show bulk index failed: 429 Too Many Requests . Scene Evidence :

kubectl -n observability logs deploy/jaeger-v2 --tail=100 | grep -iE '429|bulk|rejected' | head -5
# Output:
error exporterhelper  Exporting failed. Dropping data.
  {"kind": "exporter", "data_type": "traces", "error": "bulk index failed: status 429, es_rejected_execution_exception"}  <-- [Key]

ES write thread pool saturated:

curl -s 'http://172.16.40.41:9200/_cat/thread_pool/write?v'
# Output:
node_name  active queue rejected completed
es-hot-01  32    1000  184203   9284120  <-- [Key] 180k rejected
es-hot-02  32    1000  192841   9182041

Check index refresh interval & shard count:

ES=http://172.16.40.41:9200/sw_segment-20261118
curl -s "${ES}/_settings" | jq '.sw_segment-20261118.settings.index | {refresh_interval, number_of_shards}'
# Output:
{"refresh_interval": "1s", "number_of_shards": "3"}  <-- [Key] write-heavy index shouldn't be 1s

Collector actual write rate:

curl -s http://172.16.40.21:8888/metrics | grep -E 'otelcol_exporter_sent_spans_total|send_failed'
# Output:
otelcol_exporter_sent_spans_total 2841203
otelcol_exporter_send_failed_spans_total 1842  <-- 2.4% loss

Root Cause : refresh_interval = 1s on write-heavy trace index causes frequent segment merges, ES write thread pool queue fills (queue 1000), returns 429. Collector's exporterhelper retries exhausted then drops data, manifesting as missing latter spans of same trace (spans distributed across batches, some batches rejected). Fix :

# 1) Increase Collector retry & queue buffer
kubectl -n observability edit cm otel-config
# Add to exporters:
exporters:
  otlp/jaeger:
    endpoint: jaeger-v2:4317
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 30s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 16
      queue_size: 10000
kubectl -n observability rollout restart deploy/jaeger-v2
# 2) Adjust ES write params (dynamic, no restart)
curl -X PUT "http://172.16.40.41:9200/sw_segment-20261118/_settings" \
  -H 'Content-Type: application/json' \
  -d '{"index":{"refresh_interval":"30s", "translog.durability":"async", "translog.sync_interval":"5s"}}'
# Output: {"acknowledged":true}
# 3) Verify 429 gone
kubectl -n observability logs deploy/jaeger-v2 --tail=200 | grep -c '429'
# Output: 0
# 4) Collection failure rate (should be 0)
rate(otelcol_exporter_send_failed_spans_total[5m]) / rate(otelcol_exporter_sent_spans_total[5m])
# Before: 0.024
# After: 0

Before/After :

ES 429 rejections/hour: 184203 -> 0

Span loss rate: 2.4% -> 0%

Trace incompleteness: 30% -> 0.3%

Index write TPS: 8200 (throttled) -> 12400

Rollback : revert refresh_interval to 1s , translog.durability to request . 429 will reappear; only use if not investigating index structure issues.

6.4 Commonality Across Three Cases

Failure layer : App context / OAP queue / Storage write

Manifestation : Trace split in two / Time window blank / Trace only half

Misdiagnosis direction : Blame SDK / Blame network / Blame UI

Correct action : Wrap thread pool / Lower sampling + increase queue / Tune ES write params

Common Conclusion : Tracing failures almost never "error out", they just "have less data". "Less data" is the most dangerous failure mode because no alerts fire; only discovered when business tries to retrospect. Therefore, broken-chain rate, sampling retention rate, write failure rate — these three metrics must be monitored at same level as availability.

Part 7: One Concise Script — Trace Quality Patrol

Single script: checks probe coverage, breakage rate, sampling deviation, write health; outputs for daily report.

#!/usr/bin/env bash
# File: /usr/local/bin/trace-health-check.sh
# Purpose: Trace quality patrol (read-only)
# Usage: trace-health-check.sh
set -euo pipefail
IFS=$'
\t'

JAEGER="${JAEGER_URL:-http://172.16.40.21:16686}"
OTEL_METRICS="${OTEL_METRICS_URL:-http://172.16.40.21:8888}/metrics"
SERVICES=("order-svc" "inventory-svc" "payment-svc" "user-svc")

check_jaeger() {
  local svc="$1" json n ok sum
  json="$(curl -s --max-time 10 "${JAEGER}/api/traces?service=${svc}&limit=30")"
  n="$(jq '.data | length' <<< "${json}")"
  if [[ "${n}" -eq 0 ]]; then
    printf '%-16s %s
' "${svc}" "EMPTY probe not reporting?"
    return
  fi
  ok="$(jq '[.data[] | select((.spans|length) > 1)] | length' <<< "${json}")"
  sum="$(jq '[.data[].spans | length] | add' <<< "${json}")"
  printf '%-16s traces=%-3s multiSpan=%-3s avgSpan=%.1f
' \
    "${svc}" "${n}" "${ok}" \
    "$(awk -v s="${sum}" -v t="${n}" 'BEGIN{printf "%.1f", s/t}')"
}

check_otel() {
  local m accepted sampled dropped rate fail sent
  m="$(curl -s --max-time 10 "${OTEL_METRICS}")"
  accepted="$(awk '/^otelcol_receiver_accepted_spans/ {s+=$2} END{printf "%d", s+0}' <<< "${m}")"
  sampled="$(awk '/tail_sampling_count_spans.*sampled/ {s+=$2} END{printf "%d", s+0}' <<< "${m}")"
  dropped="$(awk '/tail_sampling_count_spans.*dropped/ {s+=$2} END{printf "%d", s+0}' <<< "${m}")"
  rate="$(awk -v s="${sampled}" -v d="${dropped}" 'BEGIN{ if (s+d==0) print "NA"; else printf "%.2f%%", 100*s/(s+d) }')"
  fail="$(awk '/^otelcol_exporter_send_failed_spans/ {s+=$2} END{printf "%d", s+0}' <<< "${m}")"
  sent="$(awk '/^otelcol_exporter_sent_spans/ {s+=$2} END{printf "%d", s+0}' <<< "${m}")"
  printf 'Received spans=%s retention=%s
' "${accepted}" "${rate}"
  printf 'Sent ok=%s failed=%s
' "${sent}" "${fail}"
}

echo "== Probe Coverage & Trace Completeness =="
for s in "${SERVICES[@]}"; do check_jaeger "${s}"; done

echo
echo "== Collector Sampling & Write =="
check_otel

Example output:

== Probe Coverage & Trace Completeness ==
order-svc        traces=30 multiSpan=30 avgSpan=9.4
inventory-svc    traces=30 multiSpan=29 avgSpan=8.1
payment-svc      traces=30 multiSpan=12 avgSpan=3.2
user-svc         EMPTY probe not reporting?  <-- [Key] coverage gap
== Collector Sampling & Write ==
Received spans=18241300 retention=5.02%
Sent ok=912344 failed=0

payment-svc avg spans only 3.2 vs others, indicating downstream services not instrumented; user-svc no data = probe missing. These findings more direct than any UI screenshot.

Part 8: Production Change Risk Matrix

Key changes with impact, pre-checks, rollback:

Lower sampling rate : may miss slow traces; pre-check: confirm tail retains errors; rollback: revert value.

Switch storage backend : historical data unqueryable; pre-check: dual-write 3 days; rollback: switch back.

Upgrade Collector : full reporting interruption; pre-check: canary 1 replica; rollback: revert image.

Add tail_sampling : memory usage up; pre-check: evaluate limit & replicas; rollback: remove processor.

Change OAP TTL : history immediately lost; pre-check: confirm audit needs; rollback: revert value.

Six-element checklist (example storage switch):

Impact: cluster-wide trace query, history irretrievable

Pre-check: dual-write 3 days, compare counts & latency

Backup: old storage retain 7 days no delete

Window: Tue-Thu 22:00-24:00

Canary: switch 1 Collector replica, observe 24h

One-click rollback: change storage.selector restart OAP

Part 9: Six Red Lines for Selection & Implementation

No sampling strategy? Don't collect full. Full collection volume per Ch3 formula usually exceeds disk budget by order of magnitude. Calculate first, then deploy — first lesson of tracing.

Broken-chain rate must be monitored. Breakage doesn't error, just makes topology "miss an edge". Without single_root metric, you never know actual coverage.

Don't mix two agents. SkyWalking Agent + OTel Agent simultaneously = bytecode conflicts manifesting as random ClassNotFound or duplicate span reporting; debugging cost extreme.

Tail sampling decision_wait must exceed P99 trace latency. P99=800ms but decision_wait =500ms causes half of slow traces judged "incomplete" and dropped — this is the other cause of "only first half" in Case 3.

Storage must be expirable. Traces are the only observability data that must TTL . recordDataTTL and ES index lifecycle policy must be configured together; configuring only one creates "index not deleted but data expired" illusion, wasting disk.

Traces must connect with logs & metrics. Trace-only systems still rely on guesswork during postmortems. traceId into logs, exemplar into metrics enables complete drill-down from dashboard to single log line.

Part 10: Summary — Tracing Value Lies in "What Questions It Answers"

Choosing SkyWalking or Jaeger depends on which questions you need to answer:

Service dependency topology? -> SkyWalking (built-in aggregation)

Which hop causes P99 latency for an endpoint? -> Both work, Jaeger more granular

Slow query linked to which SQL? -> Jaeger + log correlation

Metric anomaly drill down to trace? -> SkyWalking + exemplar

Unify with OTel ecosystem? -> Jaeger v2 (Collector native)

Don't want to run ES? -> SkyWalking + BanyanDB

Recommended rollout roadmap:

Week 1 : Instrument one service, connect trace with log correlation, verify traceId searchable in logs.

Week 2 : Calculate sampling rate via formula, integrate tail sampling, establish broken-chain & retention monitoring.

Weeks 3-4 : Roll out probes by service batches, use patrol script to find uncovered services (the EMPTY line).

Month 2 : Integrate metric exemplars & alerts, turn "view traces" into "jump from alert straight to trace".

The real barrier to distributed tracing was never installing agents, but ensuring every collected trace is actually queryable when needed . Get sampling, breakage, storage right — then it's a troubleshooting tool; get them wrong — it's just data occupying disk.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

ElasticsearchobservabilityKubernetesOpenTelemetryClickHousecapacity planningdistributed tracingtroubleshootingsamplingSkyWalkingJaegertail samplingBanyanDB
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.