How to Choose the Right Full‑Link Tracing Tool: Zipkin vs Pinpoint vs SkyWalking

Problem Background

With the rise of micro‑service architectures, a single request often traverses many services deployed across multiple servers and data centers, making it difficult to understand system behavior and diagnose performance issues.

Full‑link monitoring components, inspired by Google Dapper, are needed to trace cross‑application calls, collect performance metrics (TPS, latency, error counts) and quickly locate faults.

Target Requirements

Probe Performance : The tracing agent must add minimal overhead to throughput, CPU and memory.

Code Intrusiveness : The solution should be non‑intrusive, requiring little or no code changes for developers.

Scalability : Collectors must scale horizontally to handle large server clusters.

Data Analysis : Provide fine‑grained, code‑level visibility to pinpoint failures and bottlenecks.

Transparency : Easy to enable/disable without modifying business code.

Topology : Automatically discover and display the full service topology.

Functional Modules of a Full‑Link Monitoring System

Instrumentation and Log Generation : Embed probes (client, server, or bidirectional) that emit traceId, spanId, timestamps, tags, etc.

Log Collection and Storage : Use agents to send logs to a collector (via HTTP, MQ, gRPC, or Thrift) and store them in databases such as Elasticsearch, HBase, or Cassandra.

Analysis and Statistics : Aggregate spans into traces, compute metrics, and support real‑time and offline analysis.

Visualization and Decision Support : Provide dashboards, alerts, and topology maps for operators.

Core Concepts

Span

A span is the basic unit of work identified by a 64‑bit ID, containing fields such as traceId, name, parentId, annotations, and a debug flag.

type Span struct {
    TraceID    int64 // unique request ID
    Name       string
    ID         int64 // span ID
    ParentID   int64 // parent span ID (null for root)
    Annotation []Annotation
    Debug      bool
}

Trace

A trace is a tree of spans that represents the entire request flow from client start to server response, uniquely identified by traceId.

Annotation

Annotations record specific events within a span, typically cs (client start), sr (server receive), ss (server send), and cr (client receive).

type Annotation struct {
    Timestamp int64
    Value     string
    Host      Endpoint
    Duration  int32
}

Example Request Flow

When a user request reaches front‑end service A, it may invoke services B and C via RPC. Service B returns immediately, while service C interacts with downstream services D and E before responding to A, which finally replies to the user. The full call chain is visualized as a trace diagram.

Overall Deployment Architecture

Agents instrument applications and generate trace logs. Logstash collects logs and forwards them to Kafka. Kafka feeds data to downstream consumers such as Storm, which aggregates metrics and stores results in Elasticsearch. Trace data is also persisted in HBase for fast lookup. The collector‑agent communication uses gRPC (SkyWalking) or Thrift/HTTP (Zipkin, Pinpoint).

Solution Comparison

The three popular APM solutions are:

Zipkin : Open‑source tracing system from Twitter, provides data collection, storage, query and UI.

Pinpoint : Large‑scale Java APM from Naver, supports deep method‑level tracing.

SkyWalking : Chinese open‑source APM, supports many middleware and frameworks.

Probe Performance

Benchmarking with a Spring‑Boot application (Tomcat, Spring MVC, Redis, MySQL) showed that SkyWalking’s probe had the smallest impact on throughput, Zipkin was moderate, and Pinpoint caused the largest reduction (e.g., throughput dropped from 1385 to 774 at 500 concurrent users). CPU and memory overhead stayed within ~10% for all three.

Collector Scalability

Zipkin: Server can consume logs via HTTP or MQ; multiple Zipkin‑Server instances can consume the same MQ topics for horizontal scaling.

SkyWalking: Supports single‑node and cluster modes; agents communicate with collectors via gRPC.

Pinpoint: Also offers single‑node and cluster deployments; agents use Thrift to send data to collectors.

Data Analysis Capability

Zipkin : Shows service‑level call chains; limited to interface‑level granularity.

SkyWalking : Provides 20+ integrations (Dubbo, OkHttp, DB, MQ); richer call‑chain details.

Pinpoint : Most comprehensive; records SQL statements, supports custom alerts, and offers fine‑grained method‑level visibility.

Transparency and Ease of Enable/Disable

Zipkin requires modifying code or libraries to add tracing calls. SkyWalking and Pinpoint use bytecode instrumentation, allowing agents to be attached at startup without code changes, making them more transparent to developers.

Topology Visualization

All three tools can render full service topology maps. Pinpoint’s UI shows the most detailed information (including DB names), while Zipkin’s topology is limited to service‑to‑service links.

Pinpoint vs. Zipkin Detailed Comparison

Differences

Pinpoint offers a complete APM stack (probe, collector, storage, UI); Zipkin focuses on collector and storage with a lighter UI.

Pinpoint’s official support is limited to Java agents; Zipkin provides client libraries for many languages (Java, Scala, Go, Python, etc.).

Pinpoint uses bytecode injection for zero‑intrusion; Zipkin’s Brave library requires explicit API calls or configuration.

Pinpoint stores data in HBase; Zipkin uses Cassandra.

Similarities

Both are based on Google Dapper’s model of spans and traces, using spanId and parentSpanId to build call trees.

Implementation Difficulty

Brave’s codebase is small and easy to understand, making custom integrations straightforward. Pinpoint’s bytecode‑injection framework is more complex, requiring deeper knowledge of Java agents and Thrift protocols.

Cost and Community

Zipkin benefits from a large community (Twitter) and extensive language support. Pinpoint’s community is smaller, and extending it to non‑Java environments involves higher effort.

Summary

For short‑term needs, Pinpoint provides powerful, non‑intrusive tracing with rich UI and extensive Java support, but its learning curve and future maintenance cost are higher. Zipkin offers easier integration across many languages and a simpler stack, making it a flexible choice for heterogeneous environments. SkyWalking balances performance and feature richness, especially for Java ecosystems.