Which Distributed Tracing Tool Wins? Zipkin vs Pinpoint vs SkyWalking Deep Dive

Problem Background

With the rise of micro‑service architectures, a single request often spans multiple services that may be written in different languages, deployed across thousands of servers and multiple data centers. To quickly locate and resolve failures, full‑link monitoring tools such as Google Dapper are needed to observe cross‑application and cross‑server interactions.

In large‑scale micro‑service systems, each front‑end request generates a complex distributed call chain, leading to problems such as rapid fault detection, impact scope determination, service dependency analysis, and performance bottleneck identification. Key performance metrics include throughput (TPS), response time, and error counts.

Full‑link performance monitoring aggregates these metrics from a global to a local view, enabling quick fault source identification and significantly reducing troubleshooting time.

1 Goal Requirements

The objectives for a full‑link monitoring component, as summarized from Google Dapper, are:

1. Probe performance overhead

The APM component must impose minimal overhead. Instrumentation introduces performance loss, so low‑cost tracing and sampling are required; highly optimized services can notice even tiny overheads, potentially forcing teams to disable tracing.

2. Code intrusiveness

The tool should be non‑intrusive or minimally intrusive, transparent to developers, and not require code changes. If tracing depends on developers manually adding code, it becomes fragile and hard to maintain.

3. Extensibility

A good tracing system must support distributed deployment, have strong scalability, and provide plugin APIs so developers can extend it for unsupported components.

4. Data analysis

Data analysis must be fast and cover as many dimensions as possible, providing timely feedback for production anomalies and avoiding the need for secondary development.

2 Functional Modules

Typical full‑link monitoring systems consist of four major modules:

1. Instrumentation and log generation

Instrumentation (or “embedding points”) can be client‑side, server‑side, or bidirectional, and must record traceId, spanId, start time, protocol, caller IP/port, service name, latency, result, error info, and extensible fields.

Performance impact must be low; high‑throughput services suffer more from logging overhead, mitigated by sampling and asynchronous logging.

2. Log collection and storage

Distributed log collection uses daemons on each machine; business processes send traces to the daemon, which forwards them upstream. Multi‑level collectors follow a pub/sub pattern for load balancing. Collected data is stored for real‑time analysis and offline aggregation, with traces of the same call chain grouped together.

Each machine runs a daemon that receives Trace data from business processes.

Collectors form a hierarchical pub/sub architecture for load balancing.

Aggregated data undergoes real‑time analysis and offline storage.

Offline analysis groups logs of the same trace to reconstruct the call chain.

3. Call‑chain analysis and statistics

Spans with the same TraceID are gathered and ordered by time to form a timeline; parent‑child relationships reconstruct the call stack. Errors or timeouts are logged with TraceID for quick lookup.

Dependency metrics include strong dependency (failure breaks main flow), high dependency (high call probability), and frequent dependency (multiple calls to the same service).

Offline analysis aggregates spans by TraceID; real‑time analysis extracts current QPS and latency without aggregation.

4. Visualization and decision support

Visual dashboards display per‑stage latency, performance analysis, and dependency optimization.

3 Google Dapper

3.1 Span

A Span is the basic work unit of a trace, identified by a 64‑bit ID and containing name, annotations, timestamps, tags, and parent ID. Root spans have no parent.

type Span struct {
    TraceID    int64 // identifies a complete request
    Name       string
    ID         int64 // span ID
    ParentID   int64 // parent span ID, null for root
    Annotation []Annotation // timestamps
    Debug      bool
}

3.2 Trace

A Trace is a tree of Spans representing a complete request lifecycle from client request to server response, uniquely identified by trace_id.

3.3 Annotation

Annotations record specific events (cs, sr, ss, cr) with timestamps and optional key‑value data.

type Annotation struct {
    Timestamp int64
    Value     string
    Host      Endpoint
    Duration  int32
}

3.4 Call example

When a user request reaches front‑end service A, it calls services B and C. B returns quickly, while C interacts with downstream services D and E before responding to A, which finally returns to the user.

4 Solution Comparison

The three mainstream APM components based on Dapper are Zipkin, Pinpoint, and SkyWalking.

Zipkin – open‑source tracing system from Twitter, collects, stores, queries, and visualizes latency data.

Pinpoint – Java‑focused large‑scale APM from Naver, provides detailed method‑level tracing.

SkyWalking – Chinese open‑source APM for Java, offers tracing, alerting, and analysis.

4.1 Probe performance

Benchmarks on a Spring‑based app (Tomcat, Spring MVC, Redis, MySQL) show SkyWalking has the smallest throughput impact, Zipkin is moderate, and Pinpoint reduces throughput noticeably at 500 concurrent users.

4.2 Collector scalability

Zipkin uses HTTP or MQ for agent‑server communication; MQ is preferred for lower impact and can be scaled horizontally. SkyWalking’s collector supports single‑node and cluster modes via gRPC. Pinpoint’s collector also supports clustering, using Thrift over UDP.

4.3 Data analysis depth

Zipkin provides service‑level call graphs; SkyWalking adds middleware and framework support for richer detail; Pinpoint records method‑level data, SQL statements, and offers extensive alerting.

4.4 Developer transparency and toggling

Zipkin requires code changes or library integration; SkyWalking and Pinpoint use byte‑code instrumentation for zero‑code intrusion, allowing easy enable/disable via configuration.

4.5 Complete topology

All three tools can auto‑detect application topology, but Pinpoint shows richer details (e.g., DB names) while Zipkin’s view is limited to service‑to‑service links.

4.6 Pinpoint vs Zipkin detailed comparison

Differences

Pinpoint offers a full APM stack (probe, collector, storage, UI); Zipkin focuses on collector and storage with a lighter UI.

Pinpoint’s Java Agent provides non‑intrusive byte‑code injection; Zipkin’s Brave library requires explicit API calls.

Pinpoint stores data in HBase; Zipkin uses Cassandra.

Similarities

Both are based on Dapper’s span‑parent model, aggregating spans into traces.

Implementation effort

Brave’s codebase is small and easy to understand; Pinpoint’s byte‑code injection requires deeper knowledge of the target framework.

5 Tracing vs Monitoring

Monitoring covers system‑level metrics (CPU, memory, network) and application‑level metrics (QPS, latency, error counts) to detect anomalies and trigger alerts. Tracing focuses on call‑chain data to analyze performance and locate issues before they surface. Both share data collection, analysis, storage, and visualization pipelines but differ in data granularity and analysis goals.