JVM GC Tuning and Live Production Troubleshooting

After covering GC algorithms, generational models, and collector types, this installment moves from theory to practice, addressing real‑world online service problems such as CPU hitting 100 %, memory growing to OOM, and unresponsive interfaces.

1. Tuning First Principle: Measure Before Optimizing

Define a clear, quantifiable goal – e.g., P99 latency < 200 ms, Full GC frequency < 1 /day, GC pause ratio < 1 %.

Measure first – use monitoring and logs to locate the real bottleneck instead of guessing.

Most applications don’t need aggressive tuning – a reasonable heap size and the right collector usually suffice; over‑tuning can increase pause times.

Many GC‑related symptoms stem from code – static maps, caches without eviction, or heavy queries cause frequent Full GC or memory leaks; adjusting parameters only masks the problem.

2. Reading GC Logs (JDK 8 vs 17 Changes)

Enable GC logging:

# JDK 8 GC log parameters (a handful of switches)
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/path/gc.log \
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=10M

From JDK 9 onward, the unified logging framework (JEP 158/271) replaces the scattered switches with a single -Xlog option:

# Unified GC logging for JDK 9+ (including 17)
-Xlog:gc*:file=/path/gc.log:time,uptime,level,tags:filecount=5,filesize=10m

When upgrading from JDK 8 to 17, old -XX:+PrintGCDetails style flags become deprecated; failing to update startup scripts can cause warnings or startup failures.

Typical JDK 8 Minor GC line:

2026-07-27T10:00:00.000+0800: 1.234: [GC (Allocation Failure) 
 [PSYoungGen: 65536K->10736K(76288K)] 65536K->10744K(251392K), 0.0123456 secs]

Key parts: timestamp, GC type and cause, young‑gen size change, total heap change, pause duration.

JDK 17 equivalent (default G1):

[1.234s][info][gc] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 65M->10M(256M) 12.345ms

When analysing logs, focus on:

Minor GC frequency and duration – high frequency may indicate a too‑small young generation.

Full GC frequency – should be rare (ideally days apart); frequent Full GC signals a serious issue.

Old‑generation growth – if it only increases and never shrinks, it is a typical memory‑leak indicator.

Single‑GC pause time – directly impacts user‑perceived latency.

3. Key Parameters and Tuning Patterns

After understanding the logs, adjust the most impactful flags.

-Xms4g -Xmx4g          # Set initial and max heap to the same value to avoid dynamic resizing.
-Xmn2g                # Young generation size (or use -XX:NewRatio).
-XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=256m  # Cap metaspace.

-XX:+UseG1GC          # Choose G1 (default on JDK 9+; must be enabled on JDK 8).
-XX:MaxGCPauseMillis=200  # Target pause for G1/ZGC.

-XX:+HeapDumpOnOutOfMemoryError   # Auto‑dump heap on OOM.
-XX:HeapDumpPath=/path/dumps/    # Dump location.

Common pitfalls:

"Bigger heap is better" – larger heaps increase Full GC pause times and can disable compressed oops beyond 32 GB.

Blindly tweaking SurvivorRatio or MaxTenuringThreshold – only change them after log‑based evidence.

Calling System.gc() in production – triggers Full GC; disable with -XX:+DisableExplicitGC.

Basic tuning loop: set a goal → enable logging/monitoring → load‑test or observe → locate bottleneck → adjust one variable → re‑measure → repeat.

4. Troubleshooting Toolbox: Command‑Line Five‑Piece Set, Arthas, and JFR

JDK tools available under $JAVA_HOME/bin:

jps – List Java processes and PIDs. Example: jps -l jstat – Show GC statistics (region usage, GC count/duration). Example: jstat -gcutil <pid> 1000 jstack – Print thread stack traces (detect deadlocks, high‑CPU threads). Example: jstack <pid> jmap – Inspect heap (histogram, dump). Examples: jmap -histo <pid> or jmap -dump:format=b,file=heap.hprof <pid> jinfo – View/modify runtime JVM flags. Example: jinfo -flags <pid> Example jstat output and columns to watch:

jstat -gcutil 9527 1000
# Columns: S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
# Example output: 0 30 68 45 95 88 120 3.456 2 0.512 3.968

Watch O (old‑gen usage), YGC / FGC (Minor/Full GC counts), and FGCT (Full GC total time). A rising FGC together with high O usually indicates a memory leak.

Heap‑dump analysis uses MAT (Eclipse Memory Analyzer Tool) to examine .hprof files, view Leak Suspects, Dominator Trees, and the reference chain to GC Roots.

Arthas (open‑source Alibaba tool) attaches to a running JVM without restart. Frequently used commands: dashboard – real‑time overview of threads, memory, GC. thread -n 3 – list top 3 CPU‑consuming threads. thread -b – detect deadlocks. jad – decompile the currently loaded class. watch / trace – monitor method arguments, return values, and latency.

Java Flight Recorder (JFR) provides near‑zero‑overhead profiling. Since JDK 11 it is free and open‑source (previously required commercial features on JDK 8).

5. Three Real‑World Scenarios

5.1 CPU Spike

Symptom: monitoring alerts show CPU near 100 % and service latency increases.

Standard four‑step CLI workflow:

# ① Identify the Java process with high CPU
top   # assume PID = 9527
# ② Find the thread with highest CPU inside that process (use -H for threads)
top -Hp 9527   # assume TID = 9550
# ③ Convert TID to hex (jstack shows thread IDs in hex)
printf "%x
" 9550   # yields 254e
# ④ Search the hex ID in the thread dump to locate the offending code
jstack 9527 | grep '0x254e' -A 30   # shows the stack of the hot thread

If the hot thread is a GC thread, the root cause is likely overly frequent GC, pointing back to memory issues.

5.2 Memory Leak

Symptom: service slows over time, eventually throws java.lang.OutOfMemoryError: Java heap space; GC logs show old‑gen only growing and Full GC becoming more frequent.

Standard process:

# ① Obtain a heap dump (auto‑generated on OOM or manual when memory is high)
jmap -dump:format=b,file=heap.hprof 9527
# ② Open the dump with MAT and examine:
#    - Leak Suspects report
#    - Dominator Tree for biggest objects
#    - Path to GC Roots for the suspect objects

Typical leak culprits:

Static collections that only add objects.

Caches without eviction policies.

ThreadLocal not removed in thread‑pool scenarios.

Unclosed resources (connections, streams, listeners).

Fixing the reference chain in code resolves the leak; tuning flags alone cannot.

5.3 Deadlock

Symptom: a feature becomes unresponsive while CPU stays low – classic deadlock.

Use jstack to detect:

jstack 9527
# Output includes a section like:
# Found one Java-level deadlock:
# =============================
# "Thread-A":
#    waiting to lock monitor ... (a com.example.OrderLock), which is held by "Thread-B"
# "Thread-B":
#    waiting to lock monitor ... (a com.example.StockLock), which is held by "Thread-A"

Arthas thread -b provides the same information in one command.

Root cause is usually inconsistent lock acquisition order; resolve by enforcing a global lock order or using tryLock(timeout) with timeout.

Conclusion

First principle: measure, then optimize; set quantifiable goals; most services work fine with proper heap size and collector.

GC logs: JDK 8’s scattered -XX:+PrintGCDetails become unified -Xlog:gc* in JDK 9+; focus on Full GC frequency and old‑gen growth as leak signals.

Key flags: -Xms/-Xmx, choose the right collector, enable -XX:+HeapDumpOnOutOfMemoryError; avoid "bigger heap is better", random SurvivorRatio tweaks, and explicit System.gc().

Toolbox: command‑line five‑piece set (jps, jstat, jstack, jmap, jinfo), MAT for heap analysis, Arthas for live diagnosis, JFR/JMC (free from JDK 11).

Three high‑frequency production scenarios – CPU spike (top → jstack), memory leak (heap dump → MAT), deadlock (jstack/Arthas) – each with a repeatable step‑by‑step workflow.