How We Built a Scalable Java‑Agent APM Platform Using Pinpoint

Background

The original cloud‑music backend monitoring system suffered from four major problems:

Trace completeness loss : Traces were recorded via SDK instrumentation, making trace integrity dependent on correct instrumentation. Asynchronous context propagation often failed, causing missing or conflicting trace data and preventing back‑tracking of exception paths.

Trace‑metric separation : Metrics were stored in a different platform, so slow‑request or slow‑SQL incidents could not be directly linked to the responsible trace.

Log‑trace correlation gap : ERROR logs could be found, but without the corresponding request trace the root cause remained hidden, especially under low‑sampling scenarios.

Difficult version upgrades : Monitoring required SDK updates in each business service, making upgrades cumbersome and slowing feature iteration.

Project Goals and Solution

After evaluating open‑source tracing tools, Pinpoint was selected because its trace model closely matched the existing architecture and it offered a flexible plugin ecosystem. The project aimed to:

Decouple monitoring from business code by deploying a Java Agent that requires no code changes.

Guarantee full‑trace integrity using asynchronous context management and a tail‑based sampling strategy for exceptions.

Integrate Prometheus‑based metric collection directly in the agent.

Enable rapid diagnosis by correlating metadata, metrics, and logs with trace IDs.

Provide automatic exception‑scene capture and on‑demand diagnostic tools (JStack, Arthas).

Architecture Overview

The platform consists of two logical components:

Agent : Injected into each Java service via bytecode manipulation. It generates and propagates trace IDs, records Prometheus metrics, captures JStack dumps on long‑running calls, and writes exception IDs to a centralized cache.

Console : Collects trace, metric, and log data from agents, stores them in a time‑series database, and provides UI visualisation, search, and analysis capabilities.

Java Agent Enhancements

1. Extended Trace Data Model

Based on Pinpoint’s Trace‑Span‑SpanEvent model, additional association and propagation fields were added to support:

Multiple downstream links (e.g., fan‑out messaging).

Asynchronous callbacks and cross‑process context transmission.

Bidirectional propagation SDK for intra‑process and inter‑process metadata.

2. Tail‑Based Exception Sampling

When an exception occurs, the agent writes the TraceId to a centralized cache (e.g., Redis). A background thread runs every 30 seconds – 1 minute, scans the full‑trace log files, and extracts the complete trace for any cached TraceId. The extracted trace is then forwarded to the collector, ensuring that even low‑sampled paths have a full view of the failure chain.

3. Prometheus Integration

The agent embeds the Prometheus Java client library. Each service exposes a /metrics endpoint; the console pulls these endpoints at a configurable interval, aggregates the data into VM‑storage, and attaches the current TraceId to each metric record. This creates a one‑to‑one mapping between a metric sample and its trace, enabling queries such as “find the trace that produced a latency spike of X ms”.

4. Automatic JStack Collection

An asynchronous listener monitors method execution times. If a call exceeds a configurable threshold (default 5 s), the listener triggers a JStack dump, tags the dump with the active TraceId and relevant metric values, and stores the dump for later correlation with the trace view.

APM Product Design

The custom UI extends Pinpoint’s default console with the following capabilities:

Link Detail Diagnosis : Visualises full call topology, latency distribution, and propagates custom fields and request parameters. Users can switch between a global view and a per‑process view.

Application Monitoring Dashboards : Aggregates HTTP, RPC, MQ, DB, and cache metrics into Grafana‑based charts with comparative and trend analysis.

Exception‑Metric Correlation : Allows searching by TraceId to drill from an abnormal metric (e.g., high latency) directly to the associated trace.

JStack Traceability : Shows both automatically collected and manually requested JStack dumps alongside the related trace and thread‑pool information.

Integrated Diagnostic Tools : Provides on‑demand access to Arthas and JStack via the agent, presenting the collected data in a unified view.

Implementation Highlights

Agent bytecode injection is performed at class load time using Pinpoint’s plugin framework; plugins for HTTP, RPC, MQ, DB, and cache automatically add trace and metric hooks.

Context propagation SDK exposes PinpointContext.put(key, value) and PinpointContext.get(key) for custom metadata.

Exception tail‑sampling cache can be backed by Redis, Memcached, or an in‑process LRU map, depending on deployment scale.

Prometheus pull interval and JStack timeout thresholds are configurable via pinpoint-agent.properties.

Conclusion

The engineered solution delivers end‑to‑end observability: full trace integrity, metric‑trace correlation, automatic crash‑site capture, and integrated diagnostic tooling, all without modifying business code. Future work includes adding log‑management, alert governance, and scenario‑based event handling to complete the service‑governance ecosystem.