Understanding Distributed Tracing Systems with 30 Illustrative Diagrams

Why Distributed Tracing Matters

In micro‑service architectures a single request often traverses many modules, middleware, and machines, making it hard to know which services, nodes, and call order are involved, and where performance problems lie.

Principles and Benefits of Distributed Tracing

Key metrics for an interface are response time (RT), abnormal responses, and the main source of latency. In monolithic setups AOP can record timestamps before and after business logic to compute RT and catch exceptions.

From Monolith to Micro‑services

When services are split across machines, locating slow modules becomes difficult. Distributed tracing solves three main pain points:

Hard to troubleshoot and long resolution cycles

Hard to reproduce specific scenarios

Hard to analyze system bottlenecks

It automatically collects data, builds a complete call chain, and visualizes component performance, enabling precise bottleneck identification.

OpenTracing Standard

OpenTracing provides a lightweight, vendor‑agnostic API that sits between applications/libraries and tracing or log‑analysis tools, allowing developers to add tracing without caring about specific implementations. Its data model consists of:

Trace : a complete request chain

Span : a single call with start and end timestamps

SpanContext : global context (e.g., traceId) passed between spans

These concepts are illustrated in the accompanying diagram.

How SkyWalking Implements Distributed Tracing

SkyWalking uses a plug‑in + javaagent approach for non‑intrusive automatic span collection. Plugins are pluggable and extensible.

Cross‑process Context Propagation

Context is transmitted via message headers (e.g., Dubbo attachment) rather than the body, ensuring it travels with the request without affecting payload.

Ensuring Globally Unique traceId

SkyWalking generates IDs locally using the Snowflake algorithm for high performance. To handle clock rollback, it records the last timestamp; if the current time is earlier, a random number is used as the traceId.

Sampling Strategy

Collecting every request would produce massive data. SkyWalking defaults to sampling three times within a three‑second window. If an upstream request is sampled, downstream services are forced to sample to keep the trace complete.

Performance Evaluation

Benchmarks at 5000 TPS show negligible CPU, memory, and latency overhead compared to a baseline. Compared with Zipkin (117 ms) and Pinpoint (201 ms) under the same load, SkyWalking achieves 22 ms response time, demonstrating superior performance and non‑intrusiveness.

Company‑Specific Practices

Our team adopts only SkyWalking’s agent for sampling, keeping existing monitoring components for data storage and visualization. Customizations include:

Forced sampling in pre‑release environments via a force_flag=true cookie injected into Dubbo attachments.

Fine‑grained group sampling that ensures each of Redis, Dubbo, MySQL, etc., gets up to three samples within three seconds.

Embedding traceId into Log4j logs by defining a custom pattern converter plugin that replaces %traceId with the actual ID.

Developing proprietary plugins for Memcached and Druid, which are not provided by SkyWalking out of the box.

Plugin Development Process

A SkyWalking plugin consists of three parts:

Plugin definition class that declares the plugin.

Instrumentation that specifies the target class and method (e.g., DubboInstrumentation for MonitorFilter.invoke).

Interceptor that implements beforeMethod and afterMethod to inject the global traceId into the invocation’s attachment.

// skywalking-plugin.def file
dubbo=org.apache.skywalking.apm.plugin.asf.dubbo.DubboInstrumentation

After packaging, the plugin enhances the target method silently, preserving the “no‑code‑intrusion” principle.

Conclusion

The article walks through the theory, standards, architecture, and practical implementation of distributed tracing, using SkyWalking as a concrete example. It shows that there is no universally best solution—only the most suitable one for a given architecture.