Diagnosing Java Performance Bottlenecks with Skywalking, Arthas and Java Agents

Introduction

Performance optimization is essential for Java developers, but pinpointing bottlenecks can be time‑consuming. In a real project, an interface with a short average response time showed occasional spikes affecting about 1% of traffic, which were hard to detect through logs alone.

Performance Optimization

Adding extensive logging in complex systems leads to code clutter and frequent restarts. Combining Skywalking with Arthas provides a better solution: Skywalking identifies long‑running interfaces, while Arthas dynamically adds logs and modifies code to verify optimizations in real time.

Principle Analysis

Skywalking and Arthas both rely on Java instrumentation via a JavaAgent and the ASM bytecode framework. Arthas also uses the Attach API, OGNL, and other technologies. The analysis covers class loading mechanisms, Java Agent operation, bytecode tools, and how these components interact.

2.1 Class Loading Mechanism

The JVM loads class files through a hierarchy of class loaders. When a class is requested, the JVM first delegates to the parent loader; if it fails, the current loader loads the class via loadClass and ultimately defineClass. The native method ClassLoader.defineClass1 performs the actual loading.

2.2 Java Agent

Since JDK 1.5, the Instrumentation API allows agents to be attached at startup using -javaagent or at runtime via the Attach API ( -agentpath / -agentlib). Arthas uses the Attach method to connect to a running JVM.

2.3 Bytecode and Related Technologies

Bytecode consists of 8‑bit hexadecimal streams, making class files compact and fast to load. Tools such as ByteBuddy (used by Skywalking) and ByteKit (used by Arthas) enable runtime bytecode manipulation.

2.4 Arthas Principle Analysis

Arthas loads its JAR dynamically into the target JVM using VirtualMachine.loadAgent, which brings in arthas-agent.jar and arthas-core.jar. It isolates its classes with a custom ArthasClassLoader, while a lightweight arthas-spy module (containing SpyAPI) bridges the application and Arthas code, allowing injected code to invoke Arthas functionality without class‑loader conflicts.

2.5 Arthas Server Implementation

The server abstracts command execution, command definitions, execution modules, and code instrumentation. Commands are categorized into observation commands (e.g., version, jvm, quit) and instrumentation commands (e.g., watch, trace).

2.6 Arthas Usage

Commonly used commands include:

watch : monitor method parameters and return values.

trace : display the call stack and time spent on each node.

monitor : track method invocation count, success/failure rates, and average response time.

job : run commands asynchronously or redirect output to files.

dashboard : view real‑time JVM metrics such as memory and CPU usage.

retransform : reload modified classes without restarting the application.

quit/exit/stop : terminate the current session or stop Arthas completely.

These commands helped locate hot spots, generate flame graphs, and resolve thread‑deadlock issues in a Dubbo‑based service.

Conclusion

Understanding the underlying principles of class loading, Java agents, and bytecode manipulation enables developers to use tools like Skywalking and Arthas effectively for performance debugging, avoiding blind reliance on logs and ensuring safe, real‑time optimizations.