Midnight NullPointerException? How Arthas Fixes It in Minutes

Why Arthas?

Traditional production debugging requires digging through logs, monitoring dashboards, guessing code, or redeploying with added logs, which can take hours. Arthas attaches to a live JVM, providing real‑time visibility into threads, class versions, method arguments, return values, exceptions, call‑chain latency, and memory snapshots—all without code changes or restarts.

Quick Installation (under 5 minutes)

Verify Java is installed: java -version Download the Arthas boot JAR:

curl -O https://arthas.aliyun.com/arthas-boot.jar

or for Windows PowerShell:

powershell -Command "(New-Object System.Net.WebClient).DownloadFile('https://arthas.aliyun.com/arthas-boot.jar','arthas-boot.jar')"

Start and attach to the target process: java -jar arthas-boot.jar Enter the displayed PID to get the [arthas@pid]$ prompt. Use --telnet-port or --http-port to change ports if needed.

Six Essential Commands (cover ~90% of incidents)

1. dashboard – System health snapshot

Run dashboard to see a live table of thread IDs, names, groups, states, and CPU usage. Focus on high %CPU , abnormal MEMORY usage, and any BLOCKED threads.

2. thread – Inspect a specific thread

Identify a hot thread (e.g., http-nio-8080-exec-5) and run thread 12 to view its stack trace with class, method, and line numbers.

3. jad – Decompile online classes

Use jad com.example.service.PaymentService to view the exact bytecode version running in production, or jad -m calculateFee com.example.service.PaymentService to focus on a single method.

4. watch – Record method entry/exit

Example to capture parameters and return values:

watch com.example.service.PaymentService pay "{params, returnObj}" -x 2

To monitor exceptions, replace returnObj with throwExp. Filters can be added, e.g., "params[0]==12345" or "cost>100". Remember to stop the watch promptly for high‑frequency methods.

5. trace – Find the slow call chain

Run trace com.example.service.OrderService createOrder to see each sub‑call with its latency. Use --cost 100 to limit output to calls longer than 100 ms.

6. heapdump – Capture a memory snapshot

Execute heapdump /tmp/heap.hprof and analyze the file locally with MAT or JProfiler. Perform this during low‑traffic periods and ensure sufficient disk space.

Advanced Tips

Web console: after starting Arthas, open http://<em>server‑ip</em>:8565 for a graphical UI with auto‑completion.

Alias long commands, e.g.,

alias watchPay='watch com.example.service.PaymentService pay "{params,returnObj}" -x 2'

then simply type watchPay.

Use OGNL expressions for flexible filtering, such as

watch com.example.service.UserService update "{params[0].age, returnObj}" -x 2 "params[0].age > 30"

Common Pitfalls

Never leave watch on high‑frequency methods for long; stop it with Ctrl+C once the issue is identified.

Run heapdump off‑peak; the dump can be several gigabytes.

Start Arthas with the same OS user as the Java process, otherwise attachment fails.

Treat Arthas as a diagnostic tool, not a permanent monitor; use Prometheus/Grafana for long‑term metrics.

Full Night‑Rescue Walkthrough

Scenario: intermittent payment‑service timeouts with only “third‑party timeout” in logs.

Step 1: dashboard shows occasional CPU spikes on payment threads.

Step 2: thread reveals the hot thread stuck in HttpClient.execute.

Step 3:

watch com.example.service.PaymentService doPay "{params, returnObj, cost}" -x 2 "cost>1000"

captures a 3000 ms call where the timeout parameter is 500 ms instead of the expected 5000 ms.

Step 4: jad decompiles the front‑end API class and shows a mismatched field name causing the wrong default value.

Step 5: The front‑end team corrects the field name and redeploys; the issue disappears in under 30 minutes.

Without Arthas, the same problem would have required hours of log hunting and code guessing.

Conclusion

Arthas is not a silver bullet, but it dramatically reduces the time and fear associated with midnight production incidents by turning opaque JVM state into readable information. Adding it to your “emergency toolbox” can turn a three‑minute diagnosis into a routine operation.