How We Cut Full GC from 10 seconds to 200 ms: A Complete JVM GC Tuning Guide

Incident Overview

At 03:17 am the monitoring panel turned red: CPU hit 100 %, request latency jumped from 50 ms to 15 s, and Full GC ran continuously for about 10 seconds, pausing the service every 10 seconds.

Evidence Collection

Dump the live heap: jmap -dump:live,format=b,file=heap_before.hprof <pid> Enable GC logging with

-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:/logs/gc.log

Analyze the heap dump with MAT, which revealed three fatal issues:

Old generation too small – heap 8 GB, old generation only 5 GB (Eden:Survivor:Old = 2 GB:2 GB:512 MB:5 GB). Result: objects promoted too often.

Large objects (>2 MB) go directly to old generation – dump contained many 2 MB+ byte[]. Result: severe old‑gen fragmentation.

Default Parallel GC used – JVM reported Using ParallelOld. Result: Full GC pause time uncontrollable.

Summary: improper heap partition + wrong GC strategy created a ticking time bomb.

First Tuning – Fix Heap Partition

Corrected heap ratios before touching GC flags. -Xms8g -Xmx8g Original ratios ( -XX:NewRatio=2) gave:

Total heap 8G
├── Young = 8G/(2+1) = 2.67G
│   ├── Eden = 2.13G
│   └── S0/S1 = 0.27G each
└── Old = 5.33G  ← too small

Changed to a 1:1 old‑to‑young split and enlarged Survivor:

-Xms8g -Xmx8g
-XX:NewRatio=1          # Old:Young = 1:1, each 4G
-XX:SurvivorRatio=6     # Eden:S0:S1 = 6:1:1
-XX:MaxTenuringThreshold=15

Result: Full GC frequency dropped from every 10 seconds to every 30 seconds and pause time fell from 10 s to 4 s.

Second Tuning – Switch to G1 GC

Parallel GC prioritises throughput and gives uncontrolled pauses; it scans the entire old generation during a Full GC, which explains the 10‑second stop‑the‑world pauses on an 8 GB heap.

G1 limits pause time by dividing the heap into ~2048 regions (1‑32 MB each) and collecting only the most garbage‑filled regions.

Traditional GC: [Young][Old] → Full GC scans whole old → 10 s pause
G1: [R1][R2]…[R2048] → each cycle collects a few hot regions → ~200 ms pause

Enable G1: -XX:+UseG1GC First Full GC after the switch took 2.1 seconds, already a big improvement.

Third Tuning – Fine‑Tune G1 Parameters

-XX:MaxGCPauseMillis=200

– tells G1 to aim for pauses ≤ 200 ms (target, not guarantee).

“I want each GC pause to stay under 200 ms.”

Effect: Full GC reduced from 2.1 s to 800 ms. -XX:InitiatingHeapOccupancyPercent=45 – starts concurrent marking when heap usage reaches 45 %, preventing the old generation from filling up before a Mixed GC.

“When heap occupancy hits 45 %, trigger concurrent marking instead of waiting for the old gen to be almost full.”

Effect: Full GC down to 400 ms and frequency from every 30 seconds to every 2 minutes. -XX:G1HeapRegionSize=16m – enlarges region size so 2 MB+ objects fit into a normal region instead of becoming Humongous, reducing management overhead. Effect: Full GC down to 280 ms. -XX:G1ReservePercent=10 – reserves 10 % of the heap as emergency space to avoid promotion failures that would otherwise trigger Full GC.

“Reserve 10 % of the heap as a safety buffer.”

Effect: Full GC down to 200 ms.

Fourth Tuning – Code‑Level Fixes

The root cause was a method that allocated a 4 MB byte array on every call (QPS 500 → 2 GB/s of large objects), filling the old generation.

public List<Order> queryOrders(String userId) {
    byte[] buffer = new byte[4 * 1024 * 1024]; // 4 MB
    // … use buffer for IO
    return orders;
}

Replaced with a ThreadLocal buffer pool to reuse the array:

private static final ThreadLocal<byte[]> BUFFER_POOL =
    ThreadLocal.withInitial(() -> new byte[4 * 1024 * 1024]);

public List<Order> queryOrders(String userId) {
    byte[] buffer = BUFFER_POOL.get(); // reuse, no new
    try {
        // business logic
    } finally {
        // ThreadLocal cleans up automatically
    }
    return orders;
}

Impact: per‑second large‑object allocation dropped by 2 GB, and old‑gen promotion speed decreased by ~80 %.

Final Results (Metrics)

Full GC frequency: from every 10 seconds to every 5 minutes (↓ 120×)

Full GC pause: from 10 seconds to 200 ms (↓ 50×)

API P99 latency: from 15 seconds to 120 ms (↓ 125×)

Minor GC pause: from 200 ms to 30 ms (↓ 7×)

GC Selection Cheat‑Sheet

Web services, heap < 4 GB → Parallel GC (default) -XX:+UseParallelGC Web services, heap 4‑8 GB → G1 GC (recommended) -XX:+UseG1GC -XX:MaxGCPauseMillis=200 Large‑heap services, heap > 8 GB → G1 or ZGC -XX:+UseZGC Ultra‑low‑latency, heap > 16 GB → ZGC -XX:+UseZGC (pause < 10 ms)

JDK 21+ IO‑intensive → ZGC (generational)

-XX:+UseZGC -XX:ZCollectionInterval=5

Pitfall Checklist

-XX:+DisableExplicitGC

: disables System.gc() but some frameworks still call it, causing OOM – don’t add it, investigate the source. -XX:NewSize: fixes young‑gen size, ignored by G1 – avoid setting it with G1. -XX:MaxGCPauseMillis=50: overly aggressive, makes G1 thrash and kills throughput – use 200‑500 ms instead. -Xmn: fixes young‑gen size, conflicts with G1’s region mechanism – don’t set under G1.

Take‑aways

Allocate the heap correctly (larger young generation, reasonable old generation).

Choose the right GC strategy (G1 for 4‑8 GB, ZGC for > 16 GB, Parallel for small heaps).

Fine‑tune core parameters ( -XX:MaxGCPauseMillis, -XX:InitiatingHeapOccupancyPercent).