Distributed Tracing in Production: SkyWalking vs Jaeger Implementation Deep Dive
This comprehensive guide covers distributed tracing implementation with SkyWalking and Jaeger, detailing trace data costs, head vs tail sampling trade-offs, architecture differences, capacity planning formulas, probe integration pitfalls, cross-language trace breakage causes, and three real-world troubleshooting cases with concrete metrics and configurations.
Part 1: The Physical Essence of Traces — Why Full Instrumentation Fails
1.1 Byte Cost of a Single Trace Determines Architectural Limits
The primary engineering contradiction in distributed tracing is not functionality but the severe asymmetry between data volume and value density. A complete trace consists of multiple spans; each span's minimum attribute set includes traceId, spanId, parentSpanId, serviceName, operationName, startTime, duration, status, several tags, and possible logs (events).
Single Span minimum serialization overhead (OTLP protobuf estimate)
traceId 16 B
spanId 8 B
parentId 8 B
name+service ~40 B
timestamps 16 B
status 4 B
basic tags ~80 B
------------------------
Total ~172 B (without business attributes)
A cross-service call chain spanning 6 services: 8-12 spans
-> Each trace ~1.6 KB ~ 2.1 KB
At 3000 QPS, full collection:
-> 3000 x 2 KB x 86400 = 518 GB/day (uncompressed)Write amplification: Jaeger writing to Elasticsearch stores each span as a document; index and _source each store a copy, actual storage ~1.8-2.5x raw data. 518 GB raw becomes 1 TB+ on disk . This is why "collect all first, optimize later" fails by week two due to disk alerts.
1.2 Sampling: Mechanistic Divergence Between Head and Tail Sampling
Head-based sampling (SDK-side decision)
client -> SDK rolls dice (rate=0.01)
-> decides traceId last 8 bits, all services in chain
inherit same sampling decision via traceId
Pros: zero extra transport, controllable cost
Cons: decision made without knowing outcome; slow requests not retained "after the fact"
Tail-based sampling (Collector-side decision)
client -> full report -> Collector caches complete trace in memory/disk
-> judges (duration > 500ms or status=ERROR)
-> only retains matching traces, discards rest
Pros: 100% retention of slow and error traces
Cons: Collector needs caching and stateful routingTail sampling is the current production standard but its cost is often underestimated: Collector must guarantee all spans of the same traceId land on the same instance (otherwise judgment sees only half a chain), so a loadbalancing exporter performing traceId consistent hashing is required upfront.
skywalking-oap x2 <-(gRPC 11800)- agent
|
+-- BanyanDB / ES storage layer (sharded)
OTel Collector (gateway layer, hash by traceId)
receiver: otlp
processors: [memory_limiter, batch, tail_sampling]
exporters: [loadbalancing(traceID) -> collector-pool]
|
collector-pool x3 (stateful, cache complete trace)
|
exporter: jaeger-storage / otlp -> ClickHouse1.3 Architectural Foundation Differences: SkyWalking vs Jaeger
SkyWalking 10 (Apache Top-Level Project)
Agent (Java/PHP/Go/nginx-lua/eBPF Rover)
| gRPC 11800 / HTTP 12800
OAP (Streaming analytics engine: produces metrics directly)
| Storage abstraction layer
Storage: BanyanDB (custom) | ES (mainstream) | MySQL (small scale)
| GraphQL 12800
UI: 8080
Jaeger v2 (Built on OTel Collector)
Agent/SDK (OTel SDK, official client libs archived)
| OTLP 4317(gRPC) / 4318(HTTP)
jaeger-v2 (Config-driven single binary)
| Storage plugins
Storage: ES | OpenSearch | Cassandra | ClickHouse | Badger
| Query 16686
UI: 16686Two decisive architectural differences:
SkyWalking includes streaming aggregation : OAP computes metrics (service SLA, topology, endpoint metrics) while receiving spans, making it a native APM without extra components.
Jaeger v2 is "Collector + storage plugins" : It only handles receive, process, write; metric aggregation relies on Prometheus (via spanmetrics connector), topology view weaker than SkyWalking.
1.4 2026 Version Prerequisites (Must Align First)
Key version statuses:
Jaeger v1 : EOL 2025-12-31 -> must migrate to v2, config changes from CLI args to YAML.
Jaeger v2 : Based on OTel Collector -> reuses entire Collector component ecosystem.
SkyWalking 10.x : Current mainline; H2 storage removed, minimum requires ES.
SkyWalking BanyanDB : GA; official recommended custom storage, eliminates ES ops.
OTel Java Agent 2.x : Jaeger official client libraries archived. H2 storage removed is critical: old tutorials using storage: h2 fail to start on 10.x; must switch to BanyanDB or Elasticsearch.
1.5 Three Iron Laws of Implementation
Sampling rate must be a calculable number , not "set 100% first". Calculation method in Chapter 3.
Spans must have unified semantic conventions , otherwise cross-language traces break ( parentSpanId loss is the most common cause of broken chains).
Storage must be rotatable ; traces are the only observability data that must expire and be deleted ; retention period directly determines capacity.
Part 2: Production Baseline — Three-Layer Topology, Versions & Capacity Assumptions
2.1 Environment Assumptions
Business scale: 42 microservices, 31 Java services (Spring Boot 3.2)
Peak QPS: total 3200, single service max 900
Nodes: K8s v1.31, 8 nodes, each 16C64G
Storage: Elasticsearch 8.15 cluster (3 hot + 2 warm)
SkyWalking: OAP 10.2, BanyanDB as backup
Jaeger: v2.1 (OTel Collector distribution)
Reporting protocol: OTLP gRPC 4317 (unified entry)
2.2 Topology & Ports
[Business Pod] --OTLP:4317--> [otel-gateway svc] --> [jaeger-v2]
| (tail_sampling) :16686 UI
+--gRPC:11800----> [skywalking-oap svc] :12800
| (GraphQL)
v
[banyandb / es]
^
|
[skywalking-ui :8080]Required ports (often missed in K8s NetworkPolicy):
4317 gRPC - OTLP trace reporting
4318 HTTP - OTLP trace HTTP reporting
11800 gRPC - SkyWalking Agent reporting
12800 HTTP - OAP GraphQL query
17912 gRPC - BanyanDB data node
16686 HTTP - Jaeger Query UI
2.3 OS & Kernel Baseline: Collector is "Memory-Sensitive"
Tail sampling turns Collector from stateless to stateful; memory and connection counts become primary bottlenecks. Baselines for 16C64G Collector nodes:
# /etc/sysctl.d/90-otel.conf
# Apply: sysctl --system
# --- gRPC long connections dense, port range & TIME_WAIT pressure ---
net.ipv4.ip_local_port_range = 10240 65000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.core.somaxconn = 8192
net.ipv4.tcp_max_syn_backlog = 8192
# --- Collector memory jitter large, prevent premature OOM ---
vm.overcommit_memory = 1
vm.swappiness = 1
vm.max_map_count = 262144
# --- Heavy UDP for DNS resolution (service discovery) ---
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432# K8s: Collector Deployment resource config (must explicitly limit)
resources:
requests:
cpu: "4"
memory: 8Gi
limits:
# Note: limits and memory_limiter hard limit must leave gap
memory: 12Giresources.limits.memoryand Collector's memory_limiter must be staggered: memory_limiter.limit set to 75% of limit, otherwise cgroup OOMKill happens before graceful data drop.
2.4 Four-Dimensional Breakdown: SkyWalking OAP Config
# config/application.yml (SkyWalking 10.2, key snippets)
# Effective: OAP restart (no hot reload)
core:
default:
# [Must change] default 128, must increase when services >200
restHost: 0.0.0.0
restPort: 12800
gRPCHost: 0.0.0.0
gRPCPort: 11800
# [Tuning] Data TTL, calculate by retention & disk
recordDataTTL: 3
metricsDataTTL: 7
storage:
selector: ${SW_STORAGE:banyandb}
banyandb:
targets: ${SW_STORAGE_BANYANDB_TARGETS:banyandb:17912}
# [Tuning] Batch write size, too large causes single batch timeout
flushInterval: 3000
receiver-trace:
default:
# [Must change] Even after sampling, may be huge; must rate-limit
sampleRate: ${SW_TRACE_SAMPLE_RATE:5000}
slowDBAccessThreshold:
${SW_SLOW_DB_THRESHOLD:default:200,mongodb:100}
agent-analyzer:
default:
# [Tuning] Slow trace threshold, align with business SLA
traceAnalysis:
segmentThreshold: ${SW_SEGMENT_THRESHOLD:500}2.5 Four-Dimensional Breakdown Table
① Must-Change Items storage.selector: banyandb or elasticsearch — default H2 removed in 10.x sampleRate: calculate via formula — default value blows queue at peak recordDataTTL: 3 days — default 3 days but often mistakenly increased restHost: explicit listen address — default 127.0.0.1 makes UI unreachable
② Coupled Items sampleRate with receiver-trace queue length ( bufferChannelSize): lower sampling rate requires smaller queue, else memory wasted. recordDataTTL with disk capacity: 3 days x daily write x storage amplification = minimum disk (see Ch3 formula). metricsDataTTL with dashboard query window: TTL < window causes "curve truncation". slowDBAccessThreshold with DBA slow query threshold: inconsistency leads to same SQL judged differently in APM vs DB.
③ Performance Tuning Items (Formulas)
Sampling rate = min(target daily data / full daily data, 1)
Target daily data = tolerable disk growth / retention days / amplification factor
Baseline: 2TB available, retain 3 days, amplification 2.0
-> Target daily = 2048GB / 3 / 2.0 = 341 GB/day
Full daily data = QPS x avg trace size x 86400
-> 3200 x 1.8KB x 86400 = 497 GB/day
-> Sampling rate = 341 / 497 = 0.686
Actual take 0.5 (leave 30% margin), i.e., sampleRate=5000
(SkyWalking unit is 1/10000)④ Absolute Red Lines (Never Change) storage.selector switch mid-way -> historical data unqueryable, requires rebuild. core.default.role (cluster mode) wrong -> OAP split-brain. recordDataTTL set to 0 -> data expires immediately and gets deleted. traceId generation logic altered -> trace breaks, cannot correlate.
Part 3: Capacity, Sampling & Retention — Three Numbers Must Close the Loop
3.1 Disk Estimation Formula (Must Calculate Before Landing)
Required disk = daily write x retention days x storage amplification x (1 + 25%)
Storage amplification:
Elasticsearch 1.8 ~ 2.5 (1 replica: 3.6 ~ 5.0)
ClickHouse 1.0 ~ 1.3
BanyanDB 0.6 ~ 1.0 (columnar compression, best)
Baseline calc (ES, 1 replica):
341 GB/day x 3 days x 4.0 x 1.25 = 5115 GB ≈ 5 TBThis result forces storage re-evaluation. BanyanDB compression advantage shines: same data needs only ~1.2 TB.
Storage comparison:
BanyanDB: compression 0.6-1.0, low ops (no JVM tuning), preferred for medium-large scale.
ClickHouse: 1.0-1.3, medium ops (needs expertise), large scale & analytical.
Elasticsearch: 1.8-2.5, high ops (index tuning), reuse existing ES cluster.
Cassandra: 2.0-3.0, medium-high ops, legacy architecture, not recommended for new builds.
3.2 Retention & Query Performance Coupling
recordDataTTL = 3means querying "7-day-old traces" always fails, but postmortems often need 7-day data. Solution is not extending TTL but tiered retention :
# Short-term: full attributes, 3 days (ES hot nodes)
# Mid-term: only ERROR & P99 slow traces, 30 days (warm nodes)
# Long-term: only metrics aggregates, 1 year (monitoring DB)Implemented via two-level exporters in Collector:
processors:
filter/keep_errors_and_slow:
error_mode: ignore
traces:
span:
- 'status.code != STATUS_CODE_ERROR and duration < 500ms'
exporters:
otlp/short_term:
endpoint: jaeger-hot:4317
otlp/medium_term:
endpoint: jaeger-warm:43173.3 Sampling Rate Verification: Don't Trust Estimates, Instrument
Check actual receive rate via SkyWalking OAP GraphQL:
curl -s -X POST http://172.16.40.11:12800/graphql \
-H 'Content-Type: application/json' \
-d '{"query":"{getClusterBrief(
duration:{start:\"2026-11-18 00\",end:\"2026-11-18 01\"})
{numOfServices}}"}' | jq .Jaeger v2 Collector internal metrics (must enable telemetry):
curl -s http://172.16.40.21:8888/metrics | \
grep -E 'otelcol_(receiver_accepted|processor_)'
# Output example:
otelcol_receiver_accepted_spans{receiver="otlp"} 18241300
otelcol_processor_tail_sampling_count_spans{decision="sampled"} 9121
otelcol_processor_tail_sampling_count_spans{decision="dropped"} 18232179sampled / (sampled + dropped)= actual retention rate. If deviation from configured 5% exceeds 50%, tail_sampling policy likely wrong (usually decision_wait too short causing incomplete traces).
Verify
decision_wait</sub> adequacy (trace completeness):</p><pre><code>curl -s http://172.16.40.21:8888/metrics | \
grep 'otelcol_processor_tail_sampling_sampling_trace_dropped'
# Output:
otelcol_processor_tail_sampling_sampling_trace_dropped_too_early 1841too_early count growing means traces not fully collected before decision_wait expires; increase from 30s to 60s.
3.4 Complete Collector Tail Sampling Config
# /etc/otel/config.yaml (Jaeger v2 / OTel Collector common)
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
max_recv_msg_size_mib: 16
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_mib: 6144 # 75% of limits
spike_limit_mib: 1024
batch:
timeout: 1s
send_batch_size: 4096
send_batch_max_size: 8192
tail_sampling:
decision_wait: 30s
num_traces: 100000
expected_new_traces_per_sec: 3000
policies:
- name: errors
type: status_code
status_code: {status_codes: [ERROR]}
- name: slow
type: latency
latency: {threshold_ms: 500}
- name: baseline
type: probabilistic
probabilistic: {sampling_percentage: 5}
exporters:
loadbalancing:
routing_key: traceID
protocol:
otlp:
tls:
insecure: true
resolver:
static:
hostnames:
- jaeger-pool-0:4317
- jaeger-pool-1:4317
- jaeger-pool-2:4317
service:
telemetry:
metrics:
level: detailed
readers:
- pull:
exporter:
prometheus:
host: 0.0.0.0
port: 8888
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, tail_sampling, batch]
exporters: [loadbalancing]Note: expected_new_traces_per_sec must match actual; used for hash table pre-allocation; too small causes frequent resizing and GC jitter.
Part 4: Probe Integration — From Three Lines of Code to Production Pitfalls
4.1 Java Services: OTel Agent vs SkyWalking Agent Coexistence
Two agents mounted simultaneously interfere (both modify bytecode, both inject context). Production must choose one , or use SkyWalking's OTLP receiving capability as unified entry. Option A: Only SkyWalking Agent (OAP native, most complete topology):
java -javaagent:/opt/skywalking-agent/skywalking-agent.jar \
-Dskywalking.agent.service_name=order-svc \
-Dskywalking.collector.backend_service=172.16.40.11:11800 \
-jar /app/order-svc.jarOption B: Only OTel Agent, report via OTLP to SkyWalking OAP (SkyWalking 10 supports OTLP receiver, but topology weaker than native):
java -javaagent:/opt/otel/opentelemetry-javaagent.jar \
-Dotel.service.name=order-svc \
-Dotel.exporter.otlp.endpoint=http://172.16.40.21:4317 \
-Dotel.exporter.otlp.protocol=grpc \
-Dotel.traces.sampler=parentbased_traceidratio \
-Dotel.traces.sampler.arg=0.05 \
-jar /app/order-svc.jarK8s injection method:
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-svc
namespace: app-prod
spec:
replicas: 6
template:
metadata:
annotations:
instrumentation.opentelemetry.io/inject-java: "true"
resource.opentelemetry.io/service.version: "1.42.0"
spec:
containers:
- name: order-svc
image: harbor.example/app/order-svc:1.42.0
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "http://otel-gateway.observability:4317"
- name: OTEL_RESOURCE_ATTRIBUTES
value: "deployment.environment=prod"Verify injection without pod restart:
kubectl -n app-prod exec deploy/order-svc -- \
sh -c 'tr "\0" "
" < /proc/1/cmdline | grep javaagent'
# Output:
-javaagent:/otel-auto-instrumentation-java/javaagent.jar4.2 Nginx: Entry Layer Trace Start Most Easily Lost
Nginx doesn't generate traceId, only propagates and acts as upstream span. Use ngx_http_opentelemetry_module or skywalking-nginx-lua to generate entry span.
# /etc/nginx/conf.d/app.conf (key snippet)
# Effective: nginx -s reload (no traffic break)
# Generate or propagate W3C traceparent
map $http_traceparent $otel_traceparent {
default $http_traceparent;
"" "-";
}
server {
listen 443 ssl;
server_name app.example.com;
location /api/ {
# If no upstream traceparent, module generates
opentelemetry_operation_name "nginx-proxy";
opentelemetry_span_attributes
"http.request.method=$request_method";
proxy_set_header traceparent $otel_traceparent;
proxy_set_header X-Request-Id $request_id;
proxy_pass http://order-svc.app-prod:8080;
}
}Test and reload:
nginx -t && nginx -s reload
# Output: nginx: /etc/nginx/nginx.conf test is successfulVerify entry span generation:
curl -s -o /dev/null \
-H 'traceparent: 00-4bf92f3577b34da6-00f067aa0ba902b7-01' \
https://app.example.com/api/orders4.3 Four Typical Causes of Cross-Language Trace Breakage
Context not propagated : downstream becomes new root span -> check if parentSpanId is empty.
Middleware unsupported : MQ/thread pool breaks chain -> check if header carries traceparent.
Async not wrapped : Future breaks chain -> check sudden span count drop.
Sampling decision inconsistent : partial spans missing -> compare sampling rate configs across services.
Most common pitfall: thread pools . Java @Async or CompletableFuture.supplyAsync don't propagate Context by default; must wrap in decorator:
// Thread pool context propagation (production must-do)
Executor wrapped = Context.taskWrapping(
Executors.newFixedThreadPool(32));Quick verification of broken traces in a service: count span distribution per traceId:
kubectl -n observability exec deploy/jaeger-v2 -- \
wget -qO- 'http://localhost:16686/api/traces?service=order-svc&limit=20' \
| jq '[.data[].spans | length] | group_by(.) | map({spans: .[0], n: length})'
# Output:
[
{"spans": 3, "n": 14},
{"spans": 8, "n": 5},
{"spans": 1, "n": 1} <-- [Key anomaly] only 1 span indicates breakage
]4.4 Sidecar vs Gateway Trade-offs
Sidecar : good isolation, independent rate limiting; but per-pod memory overhead; suits high-traffic single service.
DaemonSet : node-level aggregation, resource efficient; but node failure affects whole slice; suits medium scale general.
Gateway : centralized tail sampling; but single-point pressure; suits need for full-view analysis.
Production practice: DaemonSet for batch processing + Gateway for tail sampling . DaemonSet only does batch and memory_limiter ; Gateway does tail_sampling , avoiding each node caching complete traces.
Part 5: High-Frequency Commands & Monitoring Instrumentation
5.1 Service Topology & Dependencies: What to Look at First
SkyWalking: query topology via GraphQL (better for scripting than UI):
curl -s -X POST http://172.16.40.11:12800/graphql \
-H 'Content-Type: application/json' \
-d '{"query":"query{getGlobalTopology(
duration:{start:\"2026-11-18 0900\",end:\"2026-11-18 1000\"})
{nodes{id name} calls{source target}}}"}' \
| jq '.data.getGlobalTopology.calls[0:3]'
# Output:
[
{"source": "bmdDU9ydGhlcjE=", "target": "b3JkZXItc3Zj"},
{"source": "b3JkZXItc3Zj", "target": "aW52ZW50b3J5"}
]SkyWalking node IDs are base64 encoded; scripts must decode for alert mapping. Jaeger: query recent slow traces by service:
curl -s "http://172.16.40.21:16686/api/traces?service=order-svc&lookback=1h&limit=20&minDuration=500ms" \
| jq -r '.data[] | [.traceID, (.spans|length), ((.spans | map(.duration)|max)/1000|floor)] | @tsv'
# Output:
4bf92f3577b34da6a3ce 12 1842 <-- 12 spans, max 1842ms
a1c2e3f4506071829304 6 920 <-- [Interpretation] few spans but slow
d91b2c3a4e5f60718293 9 7425.2 Key Metrics: Must-Have Dashboard Indicators
# 1) Service-level P99 latency (SkyWalking exported to Prometheus)
histogram_quantile(0.99,
sum by (le, service) (
rate(skywalking_service_latency_bucket[5m]))) > 0.5
# 2) Collector dropped span rate (>0 means rate limiting triggered)
rate(otelcol_processor_dropped_spans_total[5m]) > 0
# 3) Tail sampling retention rate (should match configured value)
sum(rate(otelcol_processor_tail_sampling_count_spans_total
{decision="sampled"}[5m]))
/ sum(rate(
otelcol_processor_tail_sampling_count_spans_total[5m]))
# 4) Collector queue usage (>80% imminent data loss)
otelcol_exporter_queue_size / otelcol_exporter_queue_capacity > 0.8# 5) Collection endpoint send failure rate
rate(otelcol_exporter_send_failed_spans_total[5m])
/ rate(otelcol_exporter_sent_spans_total[5m]) > 0.01
# 6) OAP write latency (exposed by SkyWalking 10)
skywalking_oap_persistence_timer_bulk_execute_latency_seconds > 5
# 7) Trace incompleteness rate (broken chain monitoring, custom)
sum(rate(trace_span_single_root_total[10m]))
/ sum(rate(trace_span_total[10m])) > 0.05Last metric trace_span_single_root_total is not official; must produce via Collector's count connector or custom processor. It is the only leading indicator for broken chains , worth investing.
5.3 One Command to Locate "Which Downstream Slowed the Whole Chain"
curl -s "http://172.16.40.21:16686/api/traces/${TID}" \
| jq -r '.data[0].spans[] | select(.duration > 100000) |
[.operationName, (.duration/1000|floor), .process.serviceName] | @tsv' | sort -k2 -nrOutput example:
/order/create 1842 order-svc
/order/query-status 1610 order-svc
/db/query order_item 1580 <-- [Key] occupies 86% of parent span
redis.get 12 order-svcInterpretation logic: Child span exceeds 80% of parent span and has no deeper children -> bottleneck is in that span's own latency (network or downstream processing), not deeper levels. Confirm which SQL (span tags often have db.statement ):
curl -s "http://172.16.40.21:16686/api/traces/${TID}" \
| jq -r '.data[0].spans[] | select(.operationName|test("db")) |
.tags[] | select(.key=="db.statement") | .value'
# Output:
SELECT * FROM order_item WHERE order_id = ? AND status IN (...)5.4 Log & Trace Correlation: Skip This, You're Half-Blind
Trace tells "where slow", logs tell "why slow". Must correlate via traceId .
<pattern>
%d{HH:mm:ss.SSS} [%thread] %-5level
[%X{traceId:-none}] %logger{36} - %msg%n
</pattern>Log output:
09:41:22.118 [http-nio-8080-exec-3] INFO
[4bf92f3577b34da6a3ce929d0e0e4736] c.e.OrderController
- order create startQuery all logs across services by traceId in Loki:
TID=$(curl -s "http://172.16.40.21:16686/api/traces?service=order-svc&limit=1" | jq -r '.data[0].traceID')
curl -sG http://172.16.40.31:3100/loki/api/v1/query \
--data-urlencode "query={namespace=\"app-prod\"} |= \"${TID}\"" \
| jq -r '.data.result[].values[][1]' | head -20
# Output:
09:41:22 INFO order create start trace=4bf92f...
09:41:22 WARN inventory lock wait 402ms trace=4bf92f...
09:41:23 INFO order create end cost=1842msinventory lock wait 402ms is info trace never sees — that's the value of log correlation.
5.5 Storage Layer Health Checks
ES indices & shards (trace indices prone to shard explosion):
curl -s 'http://172.16.40.41:9200/_cat/indices/sw_*?v' | head -8
# Output:
health status index pri rep docs.count store.size
green open sw_segment-20261118 3 1 38410233 41.2gb
green open sw_metrics-20261118 2 1 12840912 12.8gb
yellow open sw_segment-20261117 3 1 41203918 44.9gb
^ [Key anomaly] yellow means replica unassignedClickHouse (Jaeger alt storage): check if writes stalled by merges:
kubectl -n observability exec -it clickhouse-0 -- \
clickhouse-client -q "
SELECT formatReadableSize(sum(bytes_on_disk)) AS disk,
count() AS parts
FROM system.parts WHERE active AND table='jaeger_spans';"
# Output:
41.2 GiB 1841 <-- [Key anomaly] parts > 300 means merge can't keep upClickHouse active parts > 300 causes query latency spike; increase async_insert or reduce insert batch frequency.
Part 6: Three Deep-Water Troubleshooting Closed Loops
6.1 Case 1: Random Trace Breakage — Root Cause: Async Thread Pool Losing Context
Symptom : order-svc topology shows order-svc -> inventory call volume only 62% of actual business; 38% traces appear as two independent traces. Business reports "occasionally" can't find complete trace. Monitoring Evidence :
# Single-span trace ratio abnormally high
sum(rate(trace_span_single_root_total{service="inventory"}[10m]))
/ sum(rate(trace_span_total{service="inventory"}[10m]))
# Rose from 0.02 to 0.38Scene Evidence :
# Pull 20 inventory traces, examine span composition
curl -s "http://172.16.40.21:16686/api/traces?service=inventory&limit=20" | \
jq -r '.data[] | [.traceID, (.spans|length), (.spans[0].operationName)] | @tsv'
# Output:
a1b2c3d4e5f60718293a 1 POST /inventory/lock <-- [Key] single span
b2c3d4e5f60718293a1b 1 POST /inventory/lock <-- [Key] single span
c3d4e5f60718293a1b2c 6 /order/create <-- normal chainCheck inventory received headers (traceparent present?):
kubectl -n app-prod logs deploy/inventory-svc --tail=50 | grep -i 'traceparent\|trace_id'
# Output:
INFO received request trace_id= (empty) <-- [Key] empty
INFO received request trace_id=4bf92f3577b34da6a3ce929d0e0e4736Check order-svc call path to inventory:
kubectl -n app-prod logs deploy/order-svc --tail=200 | grep -iE 'async|thread|inventory lock' | head -5
# Output:
INFO [http-nio-8080-exec-7] async submit inventory lock task
INFO [pool-3-thread-2] inventory lock start
^ [Key] thread pool thread, context lostRoot Cause Deduction : order-svc uses @Async to call inventory asynchronously; Spring's default SimpleAsyncTaskExecutor doesn't propagate Context (i.e., traceparent ), so downstream missing parent context creates a new root span , splitting trace in two. 38% breakage matches async order proportion. Fix :
// 1) Wrap thread pool with Context.taskWrapping
@Bean("inventoryExecutor")
public Executor inventoryExecutor() {
ThreadPoolTaskExecutor ex = new ThreadPoolTaskExecutor();
ex.setCorePoolSize(32);
ex.setMaxPoolSize(64);
ex.setQueueCapacity(2000);
ex.setTaskDecorator(runnable -> Context.current().wrap(runnable));
ex.initialize();
return ex;
}# 2) Canary deploy 1 pod, verify breakage ratio
kubectl -n app-prod rollout restart deploy/order-svc
kubectl -n app-prod rollout status deploy/order-svc# 3) Compare metrics
sum(rate(trace_span_single_root_total{service="inventory"}[10m]))
/ sum(rate(trace_span_total{service="inventory"}[10m]))
# Before fix: 0.38
# After fix: 0.004Before/After Comparison :
Inventory single-span ratio: 38% -> 0.4%
Topology edge call coverage: 62% -> 99.6%
Avg spans per trace: 5.2 -> 9.4
Rollback : kubectl -n app-prod rollout undo deploy/order-svc . Change only affects context propagation; rollback introduces no data issues.
6.2 Case 2: OAP Queue Full, Massive Trace Drops
Symptom : Peak (09:00-10:00) SkyWalking UI shows topology gaps, segment data missing. Business reports traces completely unqueryable for that period; OAP logs show massive buffer is full . Scene Evidence :
kubectl -n observability logs deploy/skywalking-oap | grep -iE 'queue|reject|overflow|slow' | head -10
# Output:
WARN DataCarrier [DataCarrier-ConsumeDriver] buffer is full, dropped 18432 rows
WARN TraceAnalyzer segment buffer full, dropped 9211
ERROR GRPCStreamService status:RESOURCE_EXHAUSTED message: buffer channel size is fullCheck OAP JVM & heap:
kubectl -n observability exec deploy/skywalking-oap -- sh -c 'jcmd 1 GC.heap_info | head -6'
# Output:
PSYoungGen total 2048000K, used 2011136K
eden space 1900544K, 99% used <-- [Key] young gen pressured
PSOldGen total 4096000K, used 4012032KConfirm frequent Full GC:
kubectl -n observability exec deploy/skywalking-oap -- sh -c 'jstat -gcutil 1 1000 5'
# Output:
S0 S1 E O M CCS YGC YGCT FGC FGCT
0.00 98.10 99.32 97.94 93.12 89.44 1842 62.1 41 38.2
^ [Key] Old gen 97%, FGC 41 timesRoot Cause : sampleRate kept default (10000 = full), receiver-trace bufferChannelSize default 20000. Peak 3200 QPS x avg 9 spans = 28800 spans/sec, production rate exceeds queue consumption, queue piles up triggering RESOURCE_EXHAUSTED , gRPC side rejects directly, data lost. Fix :
# 1) Lower sampling rate first, emergency stop (config + restart OAP)
kubectl -n observability set env deploy/skywalking-oap SW_TRACE_SAMPLE_RATE=3000
kubectl -n observability rollout status deploy/skywalking-oap# 2) Increase queue & heap, combined with lower sampling
kubectl -n observability set env deploy/skywalking-oap SW_RECEIVER_BUFFER_CHANNEL_SIZE=60000
kubectl -n observability patch deploy skywalking-oap --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/limits/memory","value":"24Gi"}]'# 3) Verify no more drops
kubectl -n observability logs deploy/skywalking-oap | grep -c 'buffer is full'
# Output: 0# 4) Confirm write latency recovered
curl -s http://172.16.40.11:17128/metrics | grep persistence_timer_bulk_execute_latency_seconds | head -3
# Output:
skywalking_oap_persistence_timer_bulk_execute_latency_seconds {quantile="0.99"} 1.82 <-- [Normal] from 42.1 down to 1.82Before/After :
Queue drops/min: 18432 -> 0
OAP write latency P99: 42.1s -> 1.82s
Full GC (10min): 41 -> 3
Trace retention: ~61% -> 100%
Rollback : kubectl -n observability set env deploy/skywalking-oap SW_TRACE_SAMPLE_RATE- clear variable to restore default. Note: rollback also requires reverting bufferChannelSize , else returns to failure state.
6.3 Case 3: Jaeger Writes ES 429, Traces Only Have First Half
Symptom : 30% of traces in Jaeger UI only have first 3-4 spans, latter half missing; Collector logs show bulk index failed: 429 Too Many Requests . Scene Evidence :
kubectl -n observability logs deploy/jaeger-v2 --tail=100 | grep -iE '429|bulk|rejected' | head -5
# Output:
error exporterhelper Exporting failed. Dropping data.
{"kind": "exporter", "data_type": "traces", "error": "bulk index failed: status 429, es_rejected_execution_exception"} <-- [Key]ES write thread pool saturated:
curl -s 'http://172.16.40.41:9200/_cat/thread_pool/write?v'
# Output:
node_name active queue rejected completed
es-hot-01 32 1000 184203 9284120 <-- [Key] 180k rejected
es-hot-02 32 1000 192841 9182041Check index refresh interval & shard count:
ES=http://172.16.40.41:9200/sw_segment-20261118
curl -s "${ES}/_settings" | jq '.sw_segment-20261118.settings.index | {refresh_interval, number_of_shards}'
# Output:
{"refresh_interval": "1s", "number_of_shards": "3"} <-- [Key] write-heavy index shouldn't be 1sCollector actual write rate:
curl -s http://172.16.40.21:8888/metrics | grep -E 'otelcol_exporter_sent_spans_total|send_failed'
# Output:
otelcol_exporter_sent_spans_total 2841203
otelcol_exporter_send_failed_spans_total 1842 <-- 2.4% lossRoot Cause : refresh_interval = 1s on write-heavy trace index causes frequent segment merges, ES write thread pool queue fills (queue 1000), returns 429. Collector's exporterhelper retries exhausted then drops data, manifesting as missing latter spans of same trace (spans distributed across batches, some batches rejected). Fix :
# 1) Increase Collector retry & queue buffer
kubectl -n observability edit cm otel-config
# Add to exporters:
exporters:
otlp/jaeger:
endpoint: jaeger-v2:4317
retry_on_failure:
enabled: true
initial_interval: 1s
max_interval: 30s
max_elapsed_time: 300s
sending_queue:
enabled: true
num_consumers: 16
queue_size: 10000
kubectl -n observability rollout restart deploy/jaeger-v2# 2) Adjust ES write params (dynamic, no restart)
curl -X PUT "http://172.16.40.41:9200/sw_segment-20261118/_settings" \
-H 'Content-Type: application/json' \
-d '{"index":{"refresh_interval":"30s", "translog.durability":"async", "translog.sync_interval":"5s"}}'
# Output: {"acknowledged":true}# 3) Verify 429 gone
kubectl -n observability logs deploy/jaeger-v2 --tail=200 | grep -c '429'
# Output: 0# 4) Collection failure rate (should be 0)
rate(otelcol_exporter_send_failed_spans_total[5m]) / rate(otelcol_exporter_sent_spans_total[5m])
# Before: 0.024
# After: 0Before/After :
ES 429 rejections/hour: 184203 -> 0
Span loss rate: 2.4% -> 0%
Trace incompleteness: 30% -> 0.3%
Index write TPS: 8200 (throttled) -> 12400
Rollback : revert refresh_interval to 1s , translog.durability to request . 429 will reappear; only use if not investigating index structure issues.
6.4 Commonality Across Three Cases
Failure layer : App context / OAP queue / Storage write
Manifestation : Trace split in two / Time window blank / Trace only half
Misdiagnosis direction : Blame SDK / Blame network / Blame UI
Correct action : Wrap thread pool / Lower sampling + increase queue / Tune ES write params
Common Conclusion : Tracing failures almost never "error out", they just "have less data". "Less data" is the most dangerous failure mode because no alerts fire; only discovered when business tries to retrospect. Therefore, broken-chain rate, sampling retention rate, write failure rate — these three metrics must be monitored at same level as availability.
Part 7: One Concise Script — Trace Quality Patrol
Single script: checks probe coverage, breakage rate, sampling deviation, write health; outputs for daily report.
#!/usr/bin/env bash
# File: /usr/local/bin/trace-health-check.sh
# Purpose: Trace quality patrol (read-only)
# Usage: trace-health-check.sh
set -euo pipefail
IFS=$'
\t'
JAEGER="${JAEGER_URL:-http://172.16.40.21:16686}"
OTEL_METRICS="${OTEL_METRICS_URL:-http://172.16.40.21:8888}/metrics"
SERVICES=("order-svc" "inventory-svc" "payment-svc" "user-svc")
check_jaeger() {
local svc="$1" json n ok sum
json="$(curl -s --max-time 10 "${JAEGER}/api/traces?service=${svc}&limit=30")"
n="$(jq '.data | length' <<< "${json}")"
if [[ "${n}" -eq 0 ]]; then
printf '%-16s %s
' "${svc}" "EMPTY probe not reporting?"
return
fi
ok="$(jq '[.data[] | select((.spans|length) > 1)] | length' <<< "${json}")"
sum="$(jq '[.data[].spans | length] | add' <<< "${json}")"
printf '%-16s traces=%-3s multiSpan=%-3s avgSpan=%.1f
' \
"${svc}" "${n}" "${ok}" \
"$(awk -v s="${sum}" -v t="${n}" 'BEGIN{printf "%.1f", s/t}')"
}
check_otel() {
local m accepted sampled dropped rate fail sent
m="$(curl -s --max-time 10 "${OTEL_METRICS}")"
accepted="$(awk '/^otelcol_receiver_accepted_spans/ {s+=$2} END{printf "%d", s+0}' <<< "${m}")"
sampled="$(awk '/tail_sampling_count_spans.*sampled/ {s+=$2} END{printf "%d", s+0}' <<< "${m}")"
dropped="$(awk '/tail_sampling_count_spans.*dropped/ {s+=$2} END{printf "%d", s+0}' <<< "${m}")"
rate="$(awk -v s="${sampled}" -v d="${dropped}" 'BEGIN{ if (s+d==0) print "NA"; else printf "%.2f%%", 100*s/(s+d) }')"
fail="$(awk '/^otelcol_exporter_send_failed_spans/ {s+=$2} END{printf "%d", s+0}' <<< "${m}")"
sent="$(awk '/^otelcol_exporter_sent_spans/ {s+=$2} END{printf "%d", s+0}' <<< "${m}")"
printf 'Received spans=%s retention=%s
' "${accepted}" "${rate}"
printf 'Sent ok=%s failed=%s
' "${sent}" "${fail}"
}
echo "== Probe Coverage & Trace Completeness =="
for s in "${SERVICES[@]}"; do check_jaeger "${s}"; done
echo
echo "== Collector Sampling & Write =="
check_otelExample output:
== Probe Coverage & Trace Completeness ==
order-svc traces=30 multiSpan=30 avgSpan=9.4
inventory-svc traces=30 multiSpan=29 avgSpan=8.1
payment-svc traces=30 multiSpan=12 avgSpan=3.2
user-svc EMPTY probe not reporting? <-- [Key] coverage gap
== Collector Sampling & Write ==
Received spans=18241300 retention=5.02%
Sent ok=912344 failed=0payment-svc avg spans only 3.2 vs others, indicating downstream services not instrumented; user-svc no data = probe missing. These findings more direct than any UI screenshot.
Part 8: Production Change Risk Matrix
Key changes with impact, pre-checks, rollback:
Lower sampling rate : may miss slow traces; pre-check: confirm tail retains errors; rollback: revert value.
Switch storage backend : historical data unqueryable; pre-check: dual-write 3 days; rollback: switch back.
Upgrade Collector : full reporting interruption; pre-check: canary 1 replica; rollback: revert image.
Add tail_sampling : memory usage up; pre-check: evaluate limit & replicas; rollback: remove processor.
Change OAP TTL : history immediately lost; pre-check: confirm audit needs; rollback: revert value.
Six-element checklist (example storage switch):
Impact: cluster-wide trace query, history irretrievable
Pre-check: dual-write 3 days, compare counts & latency
Backup: old storage retain 7 days no delete
Window: Tue-Thu 22:00-24:00
Canary: switch 1 Collector replica, observe 24h
One-click rollback: change storage.selector restart OAP
Part 9: Six Red Lines for Selection & Implementation
No sampling strategy? Don't collect full. Full collection volume per Ch3 formula usually exceeds disk budget by order of magnitude. Calculate first, then deploy — first lesson of tracing.
Broken-chain rate must be monitored. Breakage doesn't error, just makes topology "miss an edge". Without single_root metric, you never know actual coverage.
Don't mix two agents. SkyWalking Agent + OTel Agent simultaneously = bytecode conflicts manifesting as random ClassNotFound or duplicate span reporting; debugging cost extreme.
Tail sampling decision_wait must exceed P99 trace latency. P99=800ms but decision_wait =500ms causes half of slow traces judged "incomplete" and dropped — this is the other cause of "only first half" in Case 3.
Storage must be expirable. Traces are the only observability data that must TTL . recordDataTTL and ES index lifecycle policy must be configured together; configuring only one creates "index not deleted but data expired" illusion, wasting disk.
Traces must connect with logs & metrics. Trace-only systems still rely on guesswork during postmortems. traceId into logs, exemplar into metrics enables complete drill-down from dashboard to single log line.
Part 10: Summary — Tracing Value Lies in "What Questions It Answers"
Choosing SkyWalking or Jaeger depends on which questions you need to answer:
Service dependency topology? -> SkyWalking (built-in aggregation)
Which hop causes P99 latency for an endpoint? -> Both work, Jaeger more granular
Slow query linked to which SQL? -> Jaeger + log correlation
Metric anomaly drill down to trace? -> SkyWalking + exemplar
Unify with OTel ecosystem? -> Jaeger v2 (Collector native)
Don't want to run ES? -> SkyWalking + BanyanDB
Recommended rollout roadmap:
Week 1 : Instrument one service, connect trace with log correlation, verify traceId searchable in logs.
Week 2 : Calculate sampling rate via formula, integrate tail sampling, establish broken-chain & retention monitoring.
Weeks 3-4 : Roll out probes by service batches, use patrol script to find uncovered services (the EMPTY line).
Month 2 : Integrate metric exemplars & alerts, turn "view traces" into "jump from alert straight to trace".
The real barrier to distributed tracing was never installing agents, but ensuring every collected trace is actually queryable when needed . Get sampling, breakage, storage right — then it's a troubleshooting tool; get them wrong — it's just data occupying disk.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
