Understanding Distributed Tracing with 40 Illustrative Diagrams

Why Distributed Tracing Matters

In a micro‑service system a single user request often traverses many modules, middle‑wares and machines. Determining which services, instances and the order of calls were involved, and locating performance bottlenecks, is difficult without a complete call chain.

Distributed Tracing Fundamentals

Three key metrics are usually monitored: response time (RT), error responses, and the slowest component. In a monolithic application developers often use AOP to log timestamps before and after business logic and to catch exceptions.

When the architecture evolves to micro‑services, the same approach no longer works because services run on different hosts. The main pain points become:

Long troubleshooting cycles.

Hard to reproduce specific scenarios.

Difficulty analyzing system bottlenecks.

Distributed tracing addresses these issues by automatically collecting data, constructing a complete Trace (the whole request), visualizing each component’s performance, and enabling precise bottleneck identification.

OpenTracing Standard

OpenTracing provides a vendor‑agnostic API that sits between applications/libraries and tracing or log‑analysis tools, similar to how JDBC abstracts database drivers. Its data model consists of:

Trace : the full request chain.

Span : a single call with start and end timestamps.

SpanContext : global context (e.g., traceId) that propagates across processes.

These concepts are illustrated in the diagram below.

How a Collector Works

The collector records for each span:

Global trace_id to link sub‑calls to the original request. span_id (e.g., 0, 1, 1.1, 2) to identify the call. parent_span_id to connect adjacent calls.

The collected data can be visualized as a call‑chain graph.

Key Challenges for a Tracing System

Automatic span collection without code intrusion.

Cross‑process context propagation.

Globally unique traceId.

Performance impact of massive data collection.

SkyWalking’s Solutions

Non‑intrusive Span Collection

SkyWalking uses a plug‑in architecture combined with a Java agent, allowing spans to be collected automatically without modifying business code.

Cross‑process Context Transfer

Context is placed in the request header (or Dubbo attachment) rather than the body, ensuring it travels with the call without affecting payload.

Globally Unique TraceId

SkyWalking generates IDs locally using the Snowflake algorithm, avoiding the latency of a central ID service. To handle the known “time‑rollback” issue, if the current timestamp is earlier than the last generated timestamp, a random number is used as the traceId.

Sampling Strategy

The default sampler collects three spans every three seconds, which can miss calls from other components (e.g., MySQL, Redis) if the first three spans are all Dubbo calls. SkyWalking was extended with:

Forced sampling in pre‑release environments via a force_flag=true cookie.

Group sampling that ensures each component type (Dubbo, Redis, MySQL) gets up to three samples within the three‑second window.

Downstream services enforce sampling when the upstream request already carries a sampling flag, guaranteeing a complete chain.

Architecture Overview

Agents collect data, which is periodically sent to storage back‑ends such as Elasticsearch or MySQL. The stored data powers visual dashboards.

Performance Evaluation

Benchmarks at 5000 TPS show negligible CPU, memory, and latency overhead when SkyWalking is enabled.

Compared with Zipkin (117 ms) and Pinpoint (201 ms) under the same load, SkyWalking’s average response time is 22 ms, demonstrating superior performance.

SkyWalking also requires no code changes (javaagent + plugins) whereas Zipkin needs explicit instrumentation, making SkyWalking less intrusive.

Company‑Specific Practices

The organization adopted only SkyWalking’s agent for sampling, keeping existing monitoring tools for storage and visualization to avoid costly migration.

Custom Enhancements

Forced sampling in pre‑release environments via a cookie flag.

Group sampling to ensure each component type gets sampled within the window.

Embedding traceId into Log4j logs by defining a custom pattern converter plugin.

Developed custom plugins for Memcached and Druid, which are not provided out‑of‑the‑box.

Plugin Development Process

A SkyWalking plugin consists of three parts:

Plugin definition class (declared in skywalking-plugin.def).

Instrumentation that specifies the target class and method.

Interceptor that implements beforeMethod and afterMethod to inject the global traceId.

For the Dubbo plugin, the MonitorFilter ’s invoke method is enhanced to insert the traceId into the invocation’s attachment before the business logic runs.

// skywalking-plugin.def file
dubbo=org.apache.skywalking.apm.plugin.asf.dubbo.DubboInstrumentation

This approach remains completely non‑intrusive to the application code.

Conclusion

The article demonstrates that distributed tracing is essential for observability in micro‑service systems, explains the underlying mechanisms of OpenTracing and SkyWalking, and shows how to tailor sampling, context propagation, and plugin development to fit specific operational needs. The key takeaway is that there is no universally “best” technology—only the most appropriate one for a given context.