How Distributed Tracing with SkyWalking Solves Microservice Performance Challenges

Principles and Benefits of Distributed Tracing Systems

In microservice architectures a single request often spans multiple modules, middleware, and machines. Determining which services, modules, and nodes are involved, as well as their call order and performance bottlenecks, requires a distributed tracing system.

Key Metrics for Interface Performance

Response time (RT)

Exception responses

Primary latency source

From Monolithic to Microservice Tracing

Monolithic applications can use simple AOP to log start and end times and capture exceptions. In microservices, the lack of a single machine makes tracing harder, leading to three main pain points: difficult problem isolation, hard‑to‑reproduce scenarios, and complex performance bottleneck analysis.

Distributed Tracing System Role

Automatic data collection

Generation of a complete call chain (Trace)

Visualization of component performance

OpenTracing Standard

OpenTracing provides a vendor‑agnostic API that sits between applications/libraries and tracing or log analysis tools, enabling interchangeable tracing implementations similar to JDBC’s driver model.

OpenTracing Data Model

Trace : a complete request chain

Span : a single call with start and end timestamps

SpanContext : global context (e.g., traceId) propagated across services

These concepts are illustrated in the following diagram:

SkyWalking Architecture and Design

Automatic Span Collection

SkyWalking uses a plugin‑based Java agent to collect spans without code intrusion. Plugins are pluggable and extensible.

Cross‑Process Context Propagation

Context is transmitted via headers (e.g., Dubbo attachment) rather than the message body, ensuring seamless propagation.

Global Unique traceId Generation

SkyWalking generates traceIds locally using the Snowflake algorithm and handles clock‑backward events by falling back to random IDs.

Sampling Strategy

To limit overhead, SkyWalking samples a few requests per interval (default 3 samples every 3 seconds). If an upstream request is sampled, downstream services force sampling to keep the trace complete.

Performance Evaluation

Benchmarks show negligible CPU, memory, and latency impact compared with no tracing, and SkyWalking outperforms Zipkin and Pinpoint in response time.

Company Practices with SkyWalking

Agent‑Only Adoption

The company uses only the SkyWalking agent for sampling, retaining existing monitoring solutions for storage and visualization.

Custom Enhancements

Forced sampling in pre‑release environments via a cookie flag.

Granular group sampling for Redis, Dubbo, MySQL, etc.

Embedding traceId into Log4j logs through a custom plugin.

Developed missing plugins for Memcached and Druid.

Plugin Implementation Overview

Each plugin consists of a definition class, instrumentation (pointcuts), and an interceptor (before/after logic). For example, the Dubbo plugin enhances the MonitorFilter.invoke method to inject the global traceId into the invocation’s attachment.

// skywalking-plugin.def
dubbo=org.apache.skywalking.apm.plugin.asf.dubbo.DubboInstrumentation

Conclusion

The article explains the principles, architecture, and practical adoption of distributed tracing with SkyWalking, emphasizing that the best technology is the one that fits the existing system rather than an absolute “best” solution.