Arthas in Production: Diagnosing Spring Boot CPU Spikes, Memory Leaks & Latency Spikes

1. Alert Scene: Check Basic Status First

The alert indicated:

Service: user-service Instance: 10.10.20.15 CPU usage: 100%, all cores saturated

Request RT: P99 jumped from 30ms to 6s

Error rate: rising rapidly, health checks failing

Logged into the server and ran top -Hp PID to see per-thread CPU usage. Many Java threads showed high CPU but the culprit wasn't obvious. Then launched Arthas.

2. Launch Arthas, Get Global View

Arthas is an open-source Java diagnostic tool from Alibaba. Download and run:

curl -O https://arthas.aliyun.com/arthas-boot.jar
java -jar arthas-boot.jar

Select the user-service process. Then run dashboard to see threads, memory, and GC overview:

ID   NAME            GROUP   PRIORITY STATE    %CPU   TIME   INTERRUPTED DAEMON
123  user-logic-1    main    5        RUNNABLE 78.0   15:32  false       false
125  user-logic-2    main    5        RUNNABLE 11.0   3:45   false       false
...
Memory          used  total  max    usage
heap            3.2G  4.0G   4.0G   80.00%
g1_old_gen      2.1G  2.5G   2.5G   84.00%
g1_young_gen    1.0G  1.5G   1.5G   66.67%

gc.g1_young_generation.count 128
gc.g1_young_generation.time  560
gc.g1_old_generation.count   12
gc.g1_old_generation.time    780

Key observations:

Thread user-logic-1 consumes 78% CPU, running for 15 minutes in RUNNABLE state.

Heap usage 3.2G/4G, old generation 84%, frequent GC.

Initial diagnosis: a business thread in a tight loop, plus memory pressure. Next, use thread command to dig deeper.

3. CPU Spike: Use thread -n 3 to Lock Busiest Threads

Typical thread usages: thread -n 3 -i 1000: sort by CPU, show top 3 busiest threads thread <id>: view full stack of a specific thread thread -b: find threads blocking others

Run thread -n 3 -i 1000. Output shows:

"user-logic-1" Id=123 RUNNABLE
  at com.example.service.UserService.processOrder(UserService.java:105)
  at com.example.service.UserService.lambda$handleRequest$0(UserService.java:89)
  at java.util.stream.StreamSpliterators$ForEachOp$OfRef.accept(StreamSpliterators.java:167)
  ...
  at java.lang.Thread.run(Thread.java:748)

Hot spot points to UserService.processOrder line 105. Source code:

public void processOrder(Order order) {
    while (order.getStatus() != OrderStatus.PAID) {
        // time-consuming status check, but status never updated
        if (retryPolicy.shouldRetry()) {
            continue; // infinite loop! no sleep, no exit condition
        }
        // other logic
    }
}

The loop spins because continue executes endlessly; retryPolicy.shouldRetry() always returns true and the order status never changes. This core business thread hogs CPU, causing request queueing and RT spike. thread -n 3 is the critical step to locate the exact line. If multiple threads stuck on same lock, use thread -b to detect deadlocks.

4. Memory Leak: Dashboard Shows Growth, Heapdump + MAT Finds Culprit

CPU issue found but memory pressure persisted. Re-run dashboard over several intervals:

Old gen usage climbed from 84% to 90%

Each Full GC reclaims little, usage rebounds immediately

Young gen allocation frequent but surviving objects remain high

Classic memory leak: objects held long-term. Arthas cannot inspect reference chains directly, but can dump heap for MAT analysis.

Dump command:

heapdump --live /tmp/user-service.hprof

--live

exports only live objects, avoiding dead noise. Ensure disk space for large hprof.

Open in Eclipse MAT. First check Histogram sorted by Retained Heap. java.util.HashMap$Node[] dominates. Follow reference chain via Path To GC Roots (exclude weak references):

com.example.cache.LocalCacheManager.INSTANCE
 -> java.util.concurrent.ConcurrentHashMap
    -> java.util.HashMap$Node[]
       -> com.example.entity.UserInfo

Root cause: LocalCacheManager uses a ConcurrentHashMap as local cache with no expiration and no capacity limit . Under high concurrency, every UserInfo (which holds address lists) gets cached, steadily consuming memory. Fix: add LRU capacity limit and TTL.

General memory leak workflow (4 steps): heapdump --live to capture snapshot

Open in MAT, Histogram sorted by Retained Heap

From largest objects, right-click Path To GC Roots, exclude weak references

Trace reference chain to business container (cache, static Map, ThreadLocal) and verify missing cleanup

5. Latency Spike: trace to See If SQL Is Slow

Scenario: CPU and memory normal, but /user/order/list P99 jumps from 100ms to 3s. Use Arthas trace and watch.

Controller method:

@RestController
public class OrderController {
    @Autowired
    private OrderService orderService;

    @GetMapping("/user/order/list")
    public List<Order> listOrders(@RequestParam String userId) {
        return orderService.queryUserOrders(userId);
    }
}

Trace OrderService.queryUserOrders:

trace com.example.service.OrderService queryUserOrders

Output per invocation:

`---[1.2s] com.example.service.OrderService.queryUserOrders()
    `---[1.0s] com.example.dao.OrderDao.selectByUserId()
        `---[0.9s] com.example.mapper.OrderMapper.selectByUserId()
            `---[0.85s] com.mysql.cj.jdbc.ConnectionImpl.prepareStatement()
                `---[0.85s] com.mysql.cj.jdbc.PreparedStatement.executeQuery()

Time spent entirely in DB query. Use watch to inspect params and result:

watch com.example.dao.OrderDao selectByUserId "{params, result}" -x 2

Params normal ( userId=12345). Check slow query log; SQL:

SELECT * FROM t_order WHERE user_id = ? AND status = 'PAID'

Table t_order has index only on user_id, not on status. MySQL scans all rows for that user then filters by status, scanning hundreds of thousands of rows as data grows.

Fix: add composite index:

ALTER TABLE t_order ADD INDEX idx_user_status (user_id, status);

To find callers of slow SQL, use stack. Other common RT spike causes: external HTTP/Redis timeouts, thread pool queue saturation, lock contention — all diagnosable via trace and thread -b.

6. Emergency Mitigation: Hot-Patch with redefine

Fixes can't wait for release. Arthas redefine swaps class bytecode in the running JVM as a stopgap.

For the dead-loop UserService.processOrder, modify code locally (add exit condition), compile with javac to .class, copy to server, then: redefine /tmp/UserService.class All instances of that class immediately use new method body. Limitations:

Cannot add/remove fields or methods; class structure immutable

Cannot change method signatures

Class name and package must match exactly

Only method bodies can change

Thus redefine is temporary only; disk JAR unchanged, changes lost on restart . Must push proper fix through CI/CD after recovery.

Note: Arthas also has vmtool to mutate object instances, but risky; avoid in production.

7. Reusable Troubleshooting Checklist

Post-incident, distilled a repeatable runbook:

Confirm symptoms : top -Hp PID for CPU/memory; check logs; health check status.

Launch Arthas : dashboard for overview; thread -n 3 -i 1000 for busiest threads; thread -b for deadlocks.

Targeted analysis :

CPU: combine stack to spot loops, regex, serialization

Memory: heapdump --live + MAT for reference chains

RT: trace to locate slow method, watch for args

SQL: slow query log + EXPLAIN for execution plan

Emergency mitigation : rate limiting, degradation, scaling; redefine for hot patch (caution); stateless services can restart.

Root cause fix : code changes, index additions, SQL optimization; monitor after deploy.

Preventive practices:

Monitoring is non-negotiable : Prometheus + Grafana for JVM threads, memory, GC, RT; APM (e.g., SkyWalking) for call traces.

Don't leave traps in code : loops must have clear exit; retries need limits and backoff; local caches require capacity bounds and eviction (Caffeine recommended); ThreadLocal must be remove d; close streams and connections.

Load test regularly : full-chain stress tests reveal capacity bottlenecks and slow SQL under concurrency.

Streamline release pipeline : slow deployments force reliance on redefine band-aids.

Summary

Most issues solvable with Arthas without restart. dashboard for global view, thread -n 3 to hit CPU hog, heapdump + MAT for memory leaks, trace / watch to dissect latency, redefine for emergency patch. Tools are powerful, but if code keeps leaving traps, Arthas can only save you so many times. Invest in monitoring and coding standards to sleep better.