Mastering Production Debugging: How Arthas Instantly Pinpoints Java Issues

Preface

I have experienced the panic of being woken up by an alarm at 3 am and the frustration of difficult-to‑locate production problems.

90% of online issues stem from the "three unknowns": not knowing which part is slow, who is blocked, and why it fails.

This article discusses how to use Arthas to quickly locate production problems.

1. Why Conventional Tools Fail Online?

Three special characteristics of production environments:

Traditional tool dilemmas:

Log loss: key parameters not logged, making post‑mortem reproduction impossible.

Monitoring lag: 1‑minute granularity misses instantaneous anomalies.

JProfiler collapse: cannot open when CPU spikes.

Arthas’s advantage:

# 1‑second attach to production environment
curl -O https://arthas.aliyun.com/arthas-boot.jar
java -jar arthas-boot.jar
# Auto‑detect Java process

2. Five Problem‑Location Scenarios

Scenario 1: Slow Interface Diagnosis

Symptom: Order query interface 99% of requests take 200 ms, 1% spike to 5 s.

Traditional solution:

// Blindly add logs
log.info("Query start: {}", System.currentTimeMillis()); // pollutes logs and inefficient

Arthas precise strike:

# 1. Trace internal method call path
trace com.example.OrderService getOrderById '#cost>1000' -n 5

Output flame graph:

Root cause: Occasional TCP connection timeout in risk control service.

Solution:

# Adjust connection timeout
risk:
  client:
    connection-timeout: 500
    read-timeout: 1000

Scenario 2: Thread Blockage Mystery

Symptom: Payment callback interface hangs at midnight.

Traditional solution:

jstack > thread.log # but blockage already ended

Arthas breakthrough:

# 1. View thread state distribution
thread -b # show blocked threads
# 2. Monitor lock competition
watch java.util.concurrent.locks.ReentrantLock getQueueLength

Diagnostic report:

Root cause: Logback synchronous logging blocks business threads.

Solution:

<appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
  <queueSize>1024</queueSize>
  <appender-ref ref="FILE"/>
</appender>

Scenario 3: Precise Memory Leak Capture

Symptom: Container restarts daily.

Traditional solution:

jmap -histo:live pid # triggers Full GC and disrupts the scene

Arthas magic:

# 1. Monitor heap objects
dashboard -i 5000 # refresh every 5 s
# 2. Trace object creation path
vmtool --action getInstances --className LoginDTO --limit 10

Detected anomaly: [LoginDTO] instances: 245,680 (growth 0.5%/min) Source code root cause:

// Bug: ThreadLocal not cleared
public class UserHolder {
  private static ThreadLocal<LoginDTO> cache = new ThreadLocal<>();
  public static void set(LoginDTO dto) { cache.set(dto); }
}

Solution:

try {
  // business code
} finally {
  UserHolder.remove(); // force cleanup
}

Scenario 4: Hot‑Fix Code Saves Crash

Symptom: New pagination query OOM, rollback takes 1 hour.

Traditional solution: Approval → merge code → compile → redeploy (causing heavy loss).

Arthas rescue:

# 1. Decompile problematic method
jad com.example.UserService listUsers
# 2. Modify local file
vi UserService.java # fix memory leak code
# 3. Hot‑update class
redefine -c 327a3b4 /tmp/UserService.class

Hot‑update principle:

Scenario 5: Data Inconsistency Mystery

Symptom: Order status shows paid, but database not updated.

Arthas investigation:

# 1. Watch method parameters/return values
watch com.service.OrderService updateStatus "{params,returnObj}" -x 3
# 2. Observe call chain
stack com.service.OrderService updateStatus

Captured abnormal call chain:

updateStatus(OrderStatus.PAID) // correct call
  |- Thread1: payment callback
updateStatus(OrderStatus.CREATED) // abnormal call
  |- Thread2: order query compensation task

Root cause: Compensation task incorrectly overwrites status.

Solution:

// Add state machine validation
if (currentStatus != CREATED) {
  throw new IllegalStateException("State rollback prohibited");
}

3. Arthas Underlying Principles

Why can it diagnose without intrusion?

Key technical breakthroughs:

Attach mechanism: uses VirtualMachine.attach to inject an agent.

Bytecode weaving: modifies method bodies with ASM to add monitoring logic.

Class isolation: custom ClassLoader prevents pollution of business code.

Diagnostic command execution flow:

4. Arthas Advanced Composite Skills

Performance analysis golden combo:

# 1. Macro overview
dashboard -i 5000
# 2. Locate CPU hotspots
profiler start # start sampling
profiler stop --format html # generate flame graph
# 3. Trace slow methods
trace *StringUtils substring '#cost>100'

Complex problem troubleshooting framework:

5. Pitfall Avoidance Guide

Three mandatory rules:

Minimization principle: watch only necessary methods, e.g., watch com.example.service.* * instead of watch * *.

Safety first: never execute high‑risk commands in production, e.g., avoid reset * and always stop when done.

Resource control: limit memory usage with options save-result false and options batch-size 50.

Conclusion

Arthas capability matrix:

Problem Type

Core Command

Effect

Method‑level tracing trace / watch Millisecond‑level performance analysis

Thread diagnosis thread / thread -b Second‑level blockage source location

Memory analysis heapdump / vmtool Memory snapshot without triggering GC

Dynamic repair jad / redefine Hot update without restart

Architect’s three‑layer realm:

Observe phenomenon: CPU high → restart (novice).

See essence: thread blockage → lock optimization (intermediate).

Envision future: chaos engineering fault injection (master).

True masters don’t just solve problems; they make problems disappear.

When you wield Arthas like a surgical knife, every online crisis becomes a stage to showcase deep technical expertise.