How a Full GC Spike Turned 50 ms P99 Latency into 3 s – Full Investigation

01 Incident: Monitoring Alarm

Core API /api/order/create normally has P99 < 50 ms. Suddenly P99 exceeded 3000 ms and many 504 Gateway Timeout responses appeared.

Normal : P99 < 50 ms, flat line.

Now : P99 > 3000 ms, many timeouts.

First reaction :

Check database – QPS normal, no slow SQL.

Check network – bandwidth only 20% utilized.

Check JVM – Full GC count surged from 1/min to 50/min.

Conclusion : Classic GC storm; CPU spent on GC, causing Stop‑The‑World pauses and latency explosion.

02 Emergency Stop‑gap: Keep Service Alive

Immediate goal is to restore service.

Restart instance : clears memory, GC frequency returns to normal, P99 drops back to 50 ms.

Preserve the incident : keep state for later root‑cause analysis.

03 Deep Investigation with Arthas + MAT

Step 1 – Identify the memory consumer

Login to the server and start Arthas: java -jar arthas-boot.jar Run dashboard and observe the first line:

ID Time CPU GC Heap NonHeap Loaded Uptime Args 1 12:00:01 100% 50 85% 15% 150 10m -Xms4g...

CPU is at 100% and GC occurs 50 times per minute – the CPU is fully consumed by GC.

Step 2 – Dump the heap

Execute heapdump to generate a dump file:

$ heapdump /tmp/heapdump.hprof Dumping heap to /tmp/heapdump.hprof ...

Step 3 – Analyze the dump with Eclipse MAT

Download heapdump.hprof locally and open it in Eclipse MAT. MAT automatically produces a “Leak Suspects” report.

Problem Suspect 1 : java.util.HashMap (Retained Heap: 3.2 GB) → com.example.service.UserService (Referrers)

The large HashMap occupies 3.2 GB and cannot be reclaimed.

04 Root‑Cause Identification: Hidden Bomb in Code

Inspecting UserService reveals the following code:

@Service
public class UserService {
    // static Map used as a cache, no expiration or size limit
    private static Map<String, User> userCache = new ConcurrentHashMap<>();

    public User getUser(String userId) {
        // 1. Check cache
        User user = userCache.get(userId);
        if (user != null) {
            return user;
        }
        // 2. Query DB
        user = userMapper.selectById(userId);
        // 3. Put into cache
        userCache.put(userId, user);
        return user;
    }
}

Issues identified:

Memory leak : the static userCache grows indefinitely as every queried user is stored forever.

GC storm : when the heap (4 GB) is almost full (e.g., cache uses 3.5 GB), the JVM is forced to perform Full GC.

Stop‑The‑World pauses : Full GC pauses all business threads; scanning a huge heap takes seconds, causing the 3‑second latency spike.

05 Solutions – From Quick Fix to Permanent Fix

Option 1: Emergency Mitigation (Configuration)

Increase heap size from 4 GB to 8 GB to give GC more breathing room.

Switch GC collector: on JDK 8 the default Parallel GC can be replaced with G1 or, on JDK 11+, with ZGC , which handle large heaps better and have shorter STW times.

-XX:+UseG1GC

Option 2: Permanent Fix (Code Change)

Introduce expiration and eviction policies.

Using Caffeine :

@Service
public class UserService {
    // Caffeine cache with max size and expiration
    private static LoadingCache<String, User> userCache = Caffeine.newBuilder()
        .maximumSize(10000)               // up to 10 k users
        .expireAfterWrite(10, TimeUnit.MINUTES) // expire after 10 min
        .build(key -> userMapper.selectById(key));

    public User getUser(String userId) {
        // Caffeine handles cache hit, load, and expiration automatically
        return userCache.get(userId);
    }
}

Or, using a plain Map with a scheduled cleanup task:

private static Map<String, User> userCache = new ConcurrentHashMap<>();
private static ScheduledExecutorService cleaner = Executors.newScheduledThreadPool(1);

// Clean up every minute
cleaner.scheduleAtFixedRate(() -> {
    long now = System.currentTimeMillis();
    userCache.entrySet().removeIf(entry -> now - entry.getValue().getUpdateTime() > 10 * 60 * 1000);
}, 1, 1, TimeUnit.MINUTES);

06 Takeaways

Static collections ( static Map, static List) are common sources of memory leaks.

Monitor trends, not just current values: watch GC frequency and GC pause time; a sudden doubling of GC count is an alarm signal.

Tools:

Online GC check: jstat -gcutil <pid> 1s Object inspection: Arthas Heap dump analysis:

Eclipse MAT