Arthas 4 Deep Dive: JDK 25, Wall-Clock Flame Graphs, and AI-Powered Diagnostics

Introduction

A three-year-old core service began slowing down daily around 3 PM. CPU and memory appeared normal, GC logs showed nothing, and monitoring metrics looked fine, yet P99 latency jumped from 50 ms to 800 ms. The team spent two days without clues. Restarting was risky and would only hide the root cause. Instead, the author attached Arthas and ran two commands: thread -n 5 This revealed a thread named pool-7-thread-3 consuming high CPU while stuck in HashMap.get(). trace on that thread's method showed a cache-key typo causing a synchronous remote call instead of a local cache hit. When the remote service slowed, the thread blocked. The entire diagnosis took under 15 minutes — no restart, no code change, no added logs.

What Is Arthas?

Arthas is a Java Agent that attaches to a running JVM via the Attach API. It requires no restart, no code changes, and no pre-instrumentation. Once attached, you interact with a shell-like terminal to diagnose the JVM. Its core capabilities fall into five categories:

Global view: dashboard (threads, memory, GC, system load), jvm (JVM parameters, class-loading stats).

Thread analysis: thread (stack traces), thread -n (top CPU threads), thread -b (deadlocks).

Method inspection: watch (parameters/return values), trace (call-chain latency), monitor (periodic stats), tt (record/replay call snapshots).

Class operations: sc (search loaded classes), jad (decompile runtime bytecode), classloader (inspect classloaders).

Hot-swapping: jad + mc + redefine to modify method bodies on the fly.

Performance profiling: profiler (flame graphs via async-profiler), jfr (JDK Flight Recorder).

What's New in Arthas 4

2.1 JDK Support: 8 to 25

Arthas 4 requires JDK 8+ and officially supports JDK 17, 21, and 25. Teams on newer runtimes no longer need to keep an old JDK just for diagnostics. Dependencies (Jackson, Fastjson2, OGNL, Netty) were also upgraded for security.

2.2 Flame Graphs: From CPU to Wall-Clock Time

The profiler command bundles async-profiler, upgraded to 4.4 in Arthas 4.2.0. Starting with 4.0.2, the --wall flag enables wall-clock profiling, which captures time spent waiting (lock contention, network I/O, database pool exhaustion) — scenarios where CPU appears low but latency is high. Traditional CPU flame graphs miss these. The default session timeout was also extended from 30 minutes to 3 hours.

2.3 MCP Server: AI-Assisted Diagnostics

Arthas 4 implements an MCP (Model Context Protocol) Server over its HTTP endpoint. AI clients (Cherry Studio, Cline, Claude Desktop) can connect and invoke 29 diagnostic tools ( dashboard, thread, trace, jad, profiler, etc.) via natural language. For example, "Show the busiest threads" triggers thread -n 5 and the AI summarizes the output. This experimental module lowers the barrier to entry — users describe symptoms instead of memorizing command syntax.

Capability Panorama

The diagram illustrates the full stack: attach → JVM → global view, method-level diagnostics, class operations, performance analysis, and MCP AI integration.

Practical Scenarios

4.1 Find the Busiest Threads

# Show top 5 CPU-consuming threads
thread -n 5

Output includes thread name, CPU usage, and current stack. If a thread is stuck in a method, follow up with trace.

4.2 Trace Method Call Latency

# Trace createOrder, only show calls > 100 ms, limit to 5 invocations
trace com.example.OrderService createOrder '#cost>100' -n 5

trace

by default traces one level of internal calls; use --maxDepth for deeper tracing (caution: high overhead). Methods inlined by JIT may not appear — this is expected, not a bug.

4.3 Observe Parameters and Return Values

# Watch params and return object, expand 2 levels
watch com.example.OrderService createOrder '{params, returnObj}' -x 2

Caveats: watch matches subclasses/implementations, potentially widening scope. On high-frequency methods, always limit with -n or conditional filters to avoid CPU spikes. Keep -x ≤ 3; expanding large objects triggers heavy reflection.

4.4 Time Tunnel (tt) for Post-Mortem Analysis

# Record invocations
tt -t com.example.OrderService createOrder
# List recordings
tt -l
# Replay the 10th invocation
tt -i 10 -p

tt

stores full call context (parameters, return value, exceptions) for later replay, even allowing parameter modification and re-execution. Critical pitfalls: default capacity is 100 records; stored object references may mutate later, showing stale values. Exiting Arthas does not auto-clear the cache — run tt --delete-all to prevent memory leaks.

4.5 Generate Flame Graphs

# CPU sampling
profiler start
profiler stop --file /tmp/cpu-flame.svg

# Wall-clock sampling (wait time)
profiler start --wall
profiler stop --file /tmp/wall-flame.svg

Open the SVG in a browser; frame width represents time consumption.

AI-Assisted Diagnostics in Practice

5.1 MCP Configuration

Enable MCP in arthas.properties:

arthas.mcpEndpoint=/mcp
arthas.httpPort=8563
arthas.password=your-secure-password

Verify with curl http://localhost:8563/mcp. Configure an AI client (e.g., Claude Desktop, Cline) with a streamable HTTP connection to that endpoint, passing the password as a Bearer token.

5.2 AI Diagnostic Scenarios

Scenario 1: CPU spike — locate hot threads. User asks: "Show the busiest threads and what they're doing." AI calls thread, returns: "Thread pool-7-thread-3 is top, stuck in HashMap.get() inside OrderService.calculateDiscount. Suggest trace on that method." Interaction completes in ~30 seconds vs. 3–8 minutes manually.

Scenario 2: Intermittent timeout — trace method latency. User: "Order submit P99 timeout; trace OrderService.submitOrder steps, focus on DB and RPC." AI runs trace com.example.OrderService submitOrder -n 10, spots inventoryClient.deduct() at 3.2 s (normal 50 ms), advises checking inventory service health and timeout config.

Scenario 3: Memory leak — heap dump. User: "Frequent old-gen GC, suspect leak; generate heap dump and analyze." AI invokes heapdump, shows memory region usage, recommends MAT analysis focusing on byte[] and HashMap retained heap.

5.3 Boundaries of AI Assistance

AI can: translate symptoms into command sequences, structure raw output into conclusions, reduce manual lookup time.

AI cannot: validate whether a conclusion matches business logic (e.g., why the inventory service timed out — network, load, or bug?). That judgment remains human.

Safety note: MCP Server is experimental. In production, restrict AI to read-only diagnostics ( trace, watch, thread); avoid letting it execute redefine or other mutating operations.

Pros and Cons

Pros

Zero-intrusion: Attaches to live JVM; no restart, no code change.

JDK 8–25 coverage: No need to retain old JDKs for diagnostics.

Complete flame graphs: async-profiler 4.4 + wall-clock sampling covers both CPU-bound and wait-bound issues.

Time Tunnel (tt): Records snapshots for post-mortem replay and parameter tweaking.

MCP + AI: 29 tools callable via natural language; promising direction despite experimental status.

Mature ecosystem: Open-sourced by Alibaba in 2018, active community, thorough documentation.

Cons

Overhead on hot methods: watch / trace add overhead; must use -n or filters on high-frequency code.

Manual tt cleanup: Cache persists after exit; forgetting tt --delete-all causes leaks.

Hot-swap limitations: Only method bodies can change; no field/method addition/removal. Conflicts with watch / trace — must reset first. Suitable only for emergency small fixes.

MCP experimental: Protocol and client support still evolving; production adoption needs caution.

Applicability

Production intermittent issues: Highly recommended — live inspection without code changes.

CPU spike diagnosis: Highly recommended — thread -n + trace quickly pinpoints hotspots.

Slow requests with low CPU: Highly recommended — profiler --wall reveals wait-time breakdown.

Class conflicts / NoSuchMethodError: Highly recommended — sc + jad + classloader resolves them.

Emergency hot-fix: Recommended — limited to method-body changes; test in staging first.

Post-mortem of rare exceptions: Highly recommended — tt captures snapshots for later replay.

AI-assisted onboarding: Recommended — MCP + AI client lowers command-learning curve.

First-time production hot-swap: Not recommended — practice in test environment; risk is high.

Conclusion

Arthas 4 transforms production diagnostics from "guessing" to "seeing" and now to "asking." You no longer rely on logs to infer thread behavior, on metrics to guess latency sources, or on redeploys to reproduce intermittent bugs. Attach Arthas, run a few commands, and the live state is visible. The 4.x roadmap is clear: track JDK evolution (now 25), complete profiling (wall-clock flame graphs), and embrace AI workflows (MCP). JDK 25 support enables use in new projects; wall-clock profiling solves "low CPU but slow" cases; MCP hints at a future where describing a symptom triggers the full diagnostic chain.

References

GitHub: https://github.com/alibaba/arthas

Official docs: https://arthas.aliyun.com