Comparing Full‑Link Tracing Tools: Zipkin vs Pinpoint vs SkyWalking

Problem Background

With the rise of micro‑service architectures, a single request often spans multiple services, which may be developed by different teams, written in various languages, and deployed across thousands of servers in multiple data centers.

Therefore, tools are needed to understand system behavior and analyze performance issues so that failures can be quickly located and resolved.

Full‑link monitoring components were created for this purpose, the most famous being Google’s Dapper.

To comprehend distributed system behavior, it is necessary to monitor cross‑application and cross‑server interactions.

In complex micro‑service systems, almost every front‑end request generates a complex distributed call chain, illustrated in the diagram below.

As business scale grows, services increase, and changes are frequent, complex call chains bring several problems:

How to quickly discover issues?

How to determine the impact range of failures?

How to map service dependencies and assess their rationality?

How to analyze link performance and plan capacity in real time?

During request processing, we also monitor performance metrics such as TPS, response time, and error records.

Through topology, calculate real‑time throughput for components, platforms, and physical devices.

Measure overall and per‑service response times.

Count error occurrences per unit time.

Full‑link performance monitoring displays metrics from global to local dimensions, consolidating cross‑application call chain information, facilitating overall and local performance measurement, and greatly reducing fault‑resolution time in production.

With a full‑link monitoring tool, we can achieve:

Request traceability for rapid fault location.

Visualization of stage durations for performance analysis.

Dependency optimization by assessing availability and relationships.

Data analysis to optimize link paths and summarize user behavior across scenarios.

1 Goal Requirements

Based on the above, the goals for a full‑link monitoring component, as mentioned in Google Dapper, include:

1. Probe Performance Overhead

The APM component should have minimal impact; instrumentation must be low‑overhead, possibly using sampling to analyze only a subset of requests.

2. Code Intrusiveness

The component should be non‑intrusive or minimally intrusive, transparent to users, and not require developers to modify application code.

3. Scalability

The tracing system must support distributed deployment and be easily extensible, offering plugin APIs for custom extensions.

4. Data Analysis

Data analysis should be fast, covering as many dimensions as possible, providing quick feedback for production anomalies.

2 Functional Modules

Typical full‑link monitoring systems consist of four major modules:

1. Instrumentation and Log Generation

Instrumentation (client, server, or bidirectional) records traceId, spanId, start time, protocol, IP/port, service name, latency, result, exception, and extensible fields.

Instrumentation must not cause performance burden; high QPS amplifies logging impact, mitigated by sampling and asynchronous logging.

2. Log Collection and Storage

Supports distributed log collection with MQ buffering. A daemon on each machine collects traces and forwards them to higher‑level collectors, which can be scaled like a pub/sub system.

Daemon collects and forwards traces.

Multi‑level collectors provide load balancing.

Real‑time analysis and offline storage of aggregated data.

Offline analysis groups logs by TraceID to reconstruct call relationships.

3. Call‑Chain Analysis and Statistics

Collect spans with the same TraceID, sort by time to form a timeline, and link ParentIDs to build the call stack. Exceptions or timeouts are logged with TraceID for quick tracing.

Dependency Metrics:

Strong dependency: failure breaks main flow.

High dependency: high probability of calling a dependency within a chain.

Frequent dependency: same dependency called many times in a chain.

Offline analysis: Aggregate by TraceID and reconstruct call relationships.

Real‑time analysis: Directly analyze individual logs to obtain current QPS and latency.

4. Presentation and Decision Support

3 Google Dapper

3.1 Span

A Span is the basic unit of a call, identified by a 64‑bit ID, containing name, parent ID, annotations, etc.

type Span struct {
    TraceID    int64 // full request ID
    Name       string
    ID         int64 // span ID
    ParentID   int64 // parent span ID, null for root
    Annotation []Annotation // timestamps
    Debug      bool
}

3.2 Trace

A Trace is a tree of Spans representing a complete request lifecycle from client request to server response.

3.3 Annotation

Annotations record specific events (e.g., timestamps) within a Span. Four standard annotations are:

cs – Client Start

sr – Server Receive

ss – Server Send

cr – Client Received

type Annotation struct {
    Timestamp int64
    Value     string
    Host      Endpoint
    Duration  int32
}

3.4 Call Example

When a user initiates a request, it first reaches front‑end service A, which calls services B and C. Service B responds to A, while C interacts with D and E before returning to A, which finally responds to the user.

4 Solution Comparison

Most full‑link monitoring models are based on Google Dapper. This article focuses on three APM components:

Zipkin – Open‑source tracing system from Twitter.

Pinpoint – Large‑scale Java APM tool from Naver.

SkyWalking – Chinese open‑source APM for Java clusters.

Comparison criteria include probe performance, collector scalability, comprehensive data analysis, developer transparency, and topology visualization.

4.1 Probe Performance

Benchmarks using a Spring‑based application (Spring Boot, MVC, Redis, MySQL) show that SkyWalking’s probe has the smallest impact on throughput, Zipkin is moderate, and Pinpoint reduces throughput significantly at 500 concurrent users.

4.2 Collector Scalability

Zipkin uses a server that can consume data via HTTP or MQ; MQ is preferred for lower impact. SkyWalking’s collector supports single‑node and cluster modes using gRPC. Pinpoint also supports single‑node and cluster deployments, communicating via Thrift.

4.3 Comprehensive Data Analysis

Zipkin provides coarse‑grained analysis at the service level. SkyWalking offers detailed analysis across 20+ middleware and frameworks. Pinpoint delivers the most complete code‑level visibility, including SQL statements and customizable alerts.

4.4 Developer Transparency and Switchability

Zipkin requires code changes via Brave libraries, while SkyWalking and Pinpoint use bytecode instrumentation, allowing deployment without modifying application code.

4.5 Full Topology Visualization

All three tools can automatically detect application topology. Pinpoint shows the richest details (e.g., DB names), Zipkin’s topology is limited to service‑to‑service links, and SkyWalking provides comprehensive views across many components.

4.6 Pinpoint vs Zipkin Detailed Comparison

Pinpoint offers a complete APM solution (probe, collector, storage, UI) whereas Zipkin focuses on collector and storage with a lighter UI. Pinpoint uses Java agents for non‑intrusive bytecode injection; Zipkin’s Brave provides API‑level instrumentation.

Pinpoint stores data in HBase, Zipkin in Cassandra. Pinpoint’s ecosystem is smaller, with fewer community plugins, while Zipkin benefits from a larger community and broader language support.

5 Tracing vs Monitoring

Monitoring includes system metrics (CPU, memory, network) and application metrics (QPS, latency, errors). Its goal is anomaly detection and alerting.

Tracing is based on call‑chain analysis, providing deeper insight for system analysis and proactive issue identification.

Both share data collection, analysis, storage, and visualization pipelines, differing mainly in the dimensions of data collected.

Source

Original source: https://www.jianshu.com/p/92a12de11f18 (DevOps技术栈)