How JD’s pfinder Achieves Full‑Stack Java Monitoring with Bytecode Magic

PFinder Overview

PFinder (Problem Finder) is a next‑generation APM system built by the UMP team. It offers call‑chain tracing, application topology, and multi‑dimensional monitoring without code changes—only two script lines added to the startup file are required. It supports JD’s mainstream middleware (jimdb, jmq, jsf) and common open‑source components (Tomcat, HTTP client, MySQL, Elasticsearch).

PFinder Features

Multi‑dimensional monitoring : statistics can be viewed by data center, group, JSF alias, caller, and other custom dimensions.

Automatic instrumentation : performance points are injected into SpringMVC, JSF, MySQL, JMQ, etc., without modifying source code.

Application topology : automatically builds upstream/downstream and middleware dependency maps.

Call‑chain tracing : cross‑service request tracing helps quickly locate performance bottlenecks.

AI‑driven fault analysis : analyzes monitoring data on the call graph to automatically determine root causes.

Traffic recording and replay : records live traffic and replays it in test or pre‑release environments to verify behavior.

Cross‑unit traffic monitoring : monitors JSF cross‑unit and escape traffic for clear visibility of modular applications.

APM Component Comparison

pfinder uniquely supports JD’s internal components such as jsf, jmq, and jimdb.

Bytecode Modification

Several mature bytecode manipulation frameworks exist, including ASM, Javassist, ByteBuddy, and ByteKit. The article demonstrates implementing the same functionality with each framework.

<code><span>@Override</span>
public void visitCode() {
    super.visitCode();
    mv.visitFieldInsn(GETSTATIC, "java/lang/System", "out", "Ljava/io/PrintStream;");
    mv.visitLdcInsn("start");
    mv.visitMethodInsn(INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/String;)V", false);
}
</code>

<code>ClassPool cp = ClassPool.getDefault();
CtClass cc = cp.get("com.ggc.javassist.HelloWord");
CtMethod m = cc.getDeclaredMethod("printHelloWord");
m.insertBefore("{ System.out.println(\"start\"); }");
m.insertAfter("{ System.out.println(\"end\"); }");
Class c = cc.toClass();
cc.writeFile("/path/to/classes");
HelloWord h = (HelloWord) c.newInstance();
h.printHelloWord();
</code>

<code>// ByteBuddy example
Class<?> dynamicType = new ByteBuddy()
    .subclass(HelloWord.class)
    .method(ElementMatchers.named("printHelloWord"))
    .intercept(MethodDelegation.to(LoggingInterceptor.class))
    .make()
    .load(HelloWord.class.getClassLoader())
    .getLoaded();
HelloWord dynamicService = (HelloWord) dynamicType.newInstance();
dynamicService.printHelloWord();
</code>

<code>public class LoggingInterceptor {
    @RuntimeType
    public static Object intercept(@AllArguments Object[] allArguments, @Origin Method method, @SuperCall Callable<?> callable) throws Exception {
        System.out.println("start");
        try {
            Object result = callable.call();
            System.out.println("end");
            return result;
        } catch (Exception e) {
            System.out.println("exception end");
            throw e;
        }
    }
}
</code>

<code>// ByteKit example (simplified)
DefaultInterceptorClassParser parser = new DefaultInterceptorClassParser();
List<InterceptorProcessor> processors = parser.parse(HelloWorldInterceptor.class);
ClassNode classNode = AsmUtils.loadClass(HelloWord.class);
for (MethodNode methodNode : classNode.methods) {
    MethodProcessor mp = new MethodProcessor(classNode, methodNode);
    for (InterceptorProcessor ip : processors) {
        ip.process(mp);
    }
}
</code>

Bytecode Injection

Debugging in IDEs like IntelliJ IDEA relies on JVMTI. The JVM specification defines JVMPI and JVMDI, which were merged into JVMTI after JDK 5. JVMTI is a native interface, but Java developers can use

java.lang.instrument.Instrumentation

to write agents in Java.

<code><archive>
    <manifestEntries>
        <!-- Specify the premain class -->
        <Agent-Class>com.ggc.agent.GhlAgent</Agent-Class>
        <Premain-Class>com.ggc.agent.GhlAgent</Premain-Class>
        <Can-Redefine-Classes>true</Can-Redefine-Classes>
        <Can-Retransform-Classes>true</Can-Retransform-Classes>
    </manifestEntries>
</archive>
</code>

<code>public class GhlAgent {
    public static Logger log = LoggerFactory.getLogger(GhlAgent.class);
    public static void agentmain(String agentArgs, Instrumentation instrumentation) {
        log.info("agentmain method");
        boot(instrumentation);
    }
    public static void premain(String agentArgs, Instrumentation instrumentation) {
        log.info("premain method");
        boot(instrumentation);
    }
    private static void boot(Instrumentation instrumentation) {
        new AgentBuilder.Default()
            .type(ElementMatchers.nameStartsWith("com.jd.aviation.performance.service.impl"))
            .transform((builder, typeDescription, classLoader, javaModule) ->
                builder.method(ElementMatchers.isMethod().and(ElementMatchers.isPublic()))
                       .intercept(MethodDelegation.to(TimingInterceptor.class)))
            .installOn(instrumentation);
    }
}
</code>

<code>public class TimingInterceptor {
    public static Logger log = LoggerFactory.getLogger(TimingInterceptor.class);
    @RuntimeType
    public static Object intercept(@SuperCall Callable<?> callable) throws Exception {
        long start = System.currentTimeMillis();
        try {
            return callable.call();
        } finally {
            long end = System.currentTimeMillis();
            log.info("Method call took {} ms", (end - start));
        }
    }
}
</code>

PFinder Architecture

When the pfinder agent starts, it loads services and plugins defined in

META-INF/pfinder/service.addon

and

META-INF/pfinder/plugin.addon

. Plugins provide bytecode enhancement points, which are applied via ByteBuddy’s

AgentBuilder

. The enhancement process involves loading services, creating matchers, and injecting advice or interceptors into target classes.

TraceId Propagation Across Threads

pfinder stores the traceId in MDC (ThreadLocal). To avoid loss when using thread pools or

@Async

, it wraps

Runnable

tasks in

TracingRunnable

, capturing the snapshot of the originating thread’s traceId and restoring it in the worker thread.

<code>public class TracingRunnable implements PfinderWrappedRunnable {
    private final Runnable origin;
    private final TracingSnapshot<?> snapshot;
    private final Component component;
    private final String operationName;
    private final String interceptorName;
    private final InterceptorClassLoader interceptorClassLoader;
    public TracingRunnable(Runnable origin, TracingSnapshot<?> snapshot, Component component, String operationName, String interceptorName, InterceptorClassLoader interceptorClassLoader) {
        this.origin = origin;
        this.snapshot = snapshot;
        this.component = component;
        this.operationName = operationName;
        this.interceptorName = interceptorName;
        this.interceptorClassLoader = interceptorClassLoader;
    }
    public void run() {
        TracingContext tracingContext = ContextManager.tracingContext();
        if (tracingContext.isTracing() && tracingContext.traceId().equals(this.snapshot.getTraceId())) {
            this.origin.run();
            return;
        }
        LowLevelAroundTracingContext context = SpringAsyncTracingContext.create(this.operationName, this.interceptorName, this.snapshot, this.interceptorClassLoader, this.component);
        context.onMethodEnter();
        try {
            this.origin.run();
        } catch (RuntimeException ex) {
            context.onException(ex);
            throw ex;
        } finally {
            context.onMethodExit();
        }
    }
    public Runnable getOrigin() { return this.origin; }
    public String toString() { return "TracingRunnable{origin=" + this.origin + ", snapshot=" + this.snapshot + ", component=" + this.component + ", operationName='" + this.operationName + "'}"; }
}
</code>

Hot Deployment

Using the javaagent, pfinder can perform class search, decompilation, and hot‑update commands sent from the server via JMT‑P. This enables online hot deployment, though limitations remain (e.g., lack of support for Spring XML, MyBatis XML, and structural class changes without DCEVM).

Future work includes extending support for configuration files and overcoming Instrumentation’s inability to modify class structure.