How Pinpoint Enables Zero‑Impact Distributed Tracing with Bytecode Injection

1. What is Pinpoint

Pinpoint is a full‑stack tracing tool that provides non‑intrusive call‑chain monitoring, method execution details, and application status monitoring, based on the Google Dapper paper.

The core idea is to record and propagate an application‑level identifier across service nodes, which is attached to HTTP headers or other protocols and used by the Pinpoint collector to link requests into a complete call chain.

Key features include:

Distributed transaction tracing across services.

Automatic detection of application topology.

Horizontal scaling for large server clusters.

Code‑level visibility to pinpoint failures and bottlenecks.

Bytecode enhancement to add functionality without modifying source code.

Pinpoint offers plugins for many components, such as JDK 6+, Tomcat, Jetty, Spring Boot, various HTTP clients, Thrift, Dubbo, databases (MySQL, Oracle, PostgreSQL, etc.), caches (Redis, Cassandra), and logging frameworks.

2. Plugin Knowledge and Data Structures

Plugins can intercept methods at the bytecode level to record execution time, parameters, return values, or inject identifiers for RPC calls. They are easy to extend, and Pinpoint provides many ready‑made plugins for common libraries.

2.1 Plugin Structure

A Pinpoint plugin consists of implementations of TraceMetadataProvider and ProfilerPlugin. The former supplies ServiceType and AnnotationKey metadata; the latter modifies target classes to collect tracing data.

Plugins are packaged as JAR files. The agent loads them via Java’s ServiceLoader mechanism, which requires provider configuration files in META-INF/services.

2.1.1 TraceMetadataProvider

Provides ServiceType definitions, which identify the library a span belongs to and dictate handling logic.

Each ServiceType includes attributes such as a unique integer code, name, and properties. The code is sent as an integer to keep payloads small, and a mapping from code to ServiceType is maintained by the provider.

Public ServiceType codes must be obtained from the Pinpoint team; private plugins can use codes from a reserved private range.

2.1.2 AnnotationKey

AnnotationKey

represents key‑value pairs attached to spans for detailed data. Built‑in keys exist, and custom keys can be added via TraceMetadataProvider.

2.2 ProfilerPlugin

The ProfilerPlugin modifies target classes to collect trace data. Its workflow:

Pinpoint Agent starts with the JVM.

Agent loads all plugins from the plugin directory.

Agent invokes each plugin’s setup(ProfilerPluginSetupContext) method.

During setup, the plugin registers class transformers.

When the application starts, classes are loaded.

For each loaded class, the agent checks for registered transformers.

If a transformer is found, the agent calls its doInTransform method.

The transformer modifies the bytecode (e.g., adds interceptors or fields).

The modified class is loaded by the JVM.

When the transformed method executes, the injected interceptor’s before and after methods run, recording trace data.

The essential steps are selecting methods to trace and injecting interceptors to capture execution details.

3. How Bytecode Injection Works

Pinpoint injects tracing code into target classes at load time using bytecode manipulation, which minimizes runtime overhead and keeps trace data compact.

Interceptors are injected around the target method, invoking before() and after() to record data.

Below is a simplified example of a custom plugin that intercepts configured methods.

First, define constants for ServiceType and AnnotationKey codes:

/* Example constants definition (image omitted) */

Define a configuration reader that parses lines like:

Method with parameters package.Clazz.MethodArgs=arg1,arg2 Method without parameters package.Clazz.MethodArgs Each line represents one configuration entry.

Implement GeneralConfigMetadataProvider to supply ServiceType metadata:

/* Metadata provider implementation (image omitted) */

Create GeneralConfigInterceptor extending SpanEventSimpleAroundInterceptorForPlugin to record method name, arguments, and return value after execution:

/* Interceptor implementation (image omitted) */

Finally, implement the plugin class that registers the interceptor with the transform template: /* Plugin registration code (image omitted) */ Add the plugin and metadata provider entries to META-INF/services:

com.navercorp.pinpoint.plugin.ProfilerPlugin--com.navercorp.pinpoint.plugin.general.config.GeneralConfigPlugin

com.navercorp.pinpoint.common.trace.TraceMetadataProvider--com.navercorp.pinpoint.plugin.general.config.GeneralConfigMetadataProvider

Deploy the JAR to the agent’s plugin directory, configure the target methods in project.properties, and start the application. The intercepted methods will appear in Pinpoint’s UI with the recorded trace information.