Master JVM Tuning on JDK 21: Reduce GC Pauses and Boost Throughput

Overview

In 2020 a production system suffered frequent Full GC pauses of up to 5 seconds, causing severe latency spikes. After expanding the heap and a two‑week systematic study of JVM tuning, the service became stable.

Technical Goals

Reduce GC frequency : fewer GC events.

Shorten GC pause : keep each pause as short as possible.

Increase throughput : maximize CPU time spent on business logic.

JDK 21 Improvements

G1 GC is the default collector and more mature.

ZGC is officially supported with a generational mode.

Virtual Threads change the concurrency model.

Applicable Scenarios

Latency‑sensitive web applications (e‑commerce, finance).

Batch processing where throughput is primary (reporting, ETL).

Large‑memory services (caches, search engines).

Low‑latency systems (trading platforms, game servers).

Environment Requirements

OS: Rocky Linux 9.4 or Ubuntu 24.04 LTS (64‑bit).

JDK: 21 LTS (Eclipse Temurin or Amazon Corretto).

Tomcat: 10.1.28 or 11.0.0 (Jakarta EE required).

Memory: ≥16 GB (≥32 GB recommended for production).

CPU: 8 cores + (GC threads scale with cores).

Preparation

Install JDK 21

# Using SDKMAN (recommended)
curl -s "https://get.sdkman.io" | bash
source "$HOME/.sdkman/bin/sdkman-init.sh"
sdk install java 21.0.4-tem

# Or manual installation
wget https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21.0.4%2B7/OpenJDK21U-jdk_x64_linux_hotspot_21.0.4_7.tar.gz
tar -xzf OpenJDK21U-jdk_x64_linux_hotspot_21.0.4_7.tar.gz -C /opt/
ln -s /opt/jdk-21.0.4+7 /opt/java

# Set environment variables
cat >> /etc/profile.d/java.sh <<'EOF'
export JAVA_HOME=/opt/java
export PATH=$JAVA_HOME/bin:$PATH
EOF
source /etc/profile.d/java.sh
java -version

Install Tomcat 10.1

# Download Tomcat
wget https://dlcdn.apache.org/tomcat/tomcat-10/v10.1.28/bin/apache-tomcat-10.1.28.tar.gz
tar -xzf apache-tomcat-10.1.28.tar.gz -C /opt/
ln -s /opt/apache-tomcat-10.1.28 /opt/tomcat

# Create tomcat user
useradd -r -s /sbin/nologin tomcat
chown -R tomcat:tomcat /opt/apache-tomcat-10.1.28

# Systemd service
cat > /etc/systemd/system/tomcat.service <<'EOF'
[Unit]
Description=Apache Tomcat Web Application Container
After=network.target

[Service]
Type=forking
User=tomcat
Group=tomcat
Environment="JAVA_HOME=/opt/java"
Environment="CATALINA_HOME=/opt/tomcat"
Environment="CATALINA_BASE=/opt/tomcat"
Environment="CATALINA_PID=/opt/tomcat/temp/tomcat.pid"
ExecStart=/opt/tomcat/bin/startup.sh
ExecStop=/opt/tomcat/bin/shutdown.sh
RestartSec=10
Restart=always

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl start tomcat
systemctl enable tomcat

Install Monitoring Tools

# jstat, jmap, jstack are included in the JDK
# Arthas (recommended)
curl -O https://arthas.aliyun.com/arthas-boot.jar
# async-profiler (performance profiling)
wget https://github.com/async-profiler/async-profiler/releases/download/v3.0/async-profiler-3.0-linux-x64.tar.gz
tar -xzf async-profiler-3.0-linux-x64.tar.gz -C /opt/

Core Configuration

JVM Memory Layout

┌─────────────────────────────────────┐
│            JVM Memory                │
├───────────────┬───────────────┤
│   Heap        │   Non‑Heap    │
│ (Young/Old)   │ (Metaspace,   │
│               │  Direct Mem) │
└───────────────┴───────────────┘

Young Generation : new objects, frequent but fast GC.

Old Generation : long‑lived objects, collected only by Full GC.

Metaspace : class metadata, replaces PermGen after JDK 8.

Direct Memory : NIO buffers, not limited by heap size.

GC Collector Selection

G1 GC (default): balances throughput and latency. Enable with -XX:+UseG1GC.

ZGC : ultra‑low latency (<1 ms pauses), suitable for large heaps. Enable with -XX:+UseZGC.

Shenandoah : similar to ZGC, RedHat development. Enable with -XX:+UseShenandoahGC.

Parallel GC : high throughput, good for batch jobs. Enable with -XX:+UseParallelGC.

Serial GC : single‑threaded, for small heaps or embedded devices. Enable with -XX:+UseSerialGC.

Recommendation : Use G1 for most cases; switch to ZGC for latency‑critical workloads; use Parallel GC when throughput dominates.

G1 GC Tuning Parameters

# /opt/tomcat/bin/setenv.sh
JAVA_OPTS="-Xms4g -Xmx4g"
JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"
JAVA_OPTS="$JAVA_OPTS -XX:MaxGCPauseMillis=200"
JAVA_OPTS="$JAVA_OPTS -XX:InitiatingHeapOccupancyPercent=45"
# Optional region size (auto‑selected for 4‑8 GB heap)
# JAVA_OPTS="$JAVA_OPTS -XX:G1HeapRegionSize=4m"
# GC logging (JDK 9+ format)
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/tomcat/gc.log:time,uptime,level,tags:filecount=10,filesize=50m"
# OOM handling
JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
JAVA_OPTS="$JAVA_OPTS -XX:HeapDumpPath=/var/log/tomcat/heapdump.hprof"
export JAVA_OPTS

Parameter details: -Xms and -Xmx should be equal to avoid dynamic heap expansion. -XX:MaxGCPauseMillis is a target, not a hard limit; G1 will try to stay below it. -XX:InitiatingHeapOccupancyPercent controls when Mixed GC starts; too low causes frequent GC, too high may trigger Full GC.

ZGC Tuning Parameters

# /opt/tomcat/bin/setenv.sh
JAVA_OPTS="-Xms8g -Xmx8g"
JAVA_OPTS="$JAVA_OPTS -XX:+UseZGC"
JAVA_OPTS="$JAVA_OPTS -XX:+ZGenerational"   # JDK 21 generational ZGC
JAVA_OPTS="$JAVA_OPTS -XX:SoftRefLRUPolicyMSPerMB=50"
JAVA_OPTS="$JAVA_OPTS -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m"
# GC logging
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:file=/var/log/tomcat/gc.log:time,uptime,level,tags:filecount=10,filesize=100m"
export JAVA_OPTS

Tomcat Thread‑Pool Tuning

<Connector port="8080" protocol="HTTP/1.1"
           connectionTimeout="20000"
           redirectPort="8443"
           maxThreads="500"
           minSpareThreads="50"
           maxConnections="10000"
           acceptCount="100"
           enableLookups="false"
           compression="on"
           compressionMinSize="1024"
           compressibleMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript"
           URIEncoding="UTF-8" />

Key parameters:

maxThreads : default 200, recommended 500 for an 8‑core server.

minSpareThreads : default 10, raise to 50 to keep idle threads ready.

maxConnections : 10000 (maximum simultaneous sockets).

acceptCount : queue length for pending connections.

Common Pitfalls

Setting maxThreads too high (e.g., 2000) can cause Full GC because each thread consumes stack memory, reducing available heap.

OptimalThreads = CPU_cores * (1 + IO_wait/CPU_compute)

For I/O‑bound web services, a multiplier of 20‑50 × CPU cores is typical.

Example Configurations

Production setenv.sh (G1, 8‑core, 16 GB RAM)

#!/bin/bash
# Memory configuration
JAVA_OPTS="-Xms8g -Xmx8g"
JAVA_OPTS="$JAVA_OPTS -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m"
JAVA_OPTS="$JAVA_OPTS -XX:MaxDirectMemorySize=1g"
# G1 GC configuration
JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"
JAVA_OPTS="$JAVA_OPTS -XX:MaxGCPauseMillis=200"
JAVA_OPTS="$JAVA_OPTS -XX:InitiatingHeapOccupancyPercent=45"
JAVA_OPTS="$JAVA_OPTS -XX:G1HeapRegionSize=4m"
JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=8"
JAVA_OPTS="$JAVA_OPTS -XX:ConcGCThreads=2"
# GC logging
GC_LOG_DIR="/var/log/tomcat"
mkdir -p $GC_LOG_DIR
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*,gc+age=debug,gc+heap=debug:file=${GC_LOG_DIR}/gc-%t.log:time,uptime,level,tags:filecount=10,filesize=100m"
# OOM handling
JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
JAVA_OPTS="$JAVA_OPTS -XX:HeapDumpPath=${GC_LOG_DIR}/heapdump-%t.hprof"
JAVA_OPTS="$JAVA_OPTS -XX:+ExitOnOutOfMemoryError"
# Performance tweaks
JAVA_OPTS="$JAVA_OPTS -XX:+DisableExplicitGC"
JAVA_OPTS="$JAVA_OPTS -XX:+UseStringDeduplication"
export JAVA_OPTS

Low‑Latency (ZGC) Configuration

#!/bin/bash
JAVA_OPTS="-Xms16g -Xmx16g"
JAVA_OPTS="$JAVA_OPTS -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m"
JAVA_OPTS="$JAVA_OPTS -XX:+UseZGC"
JAVA_OPTS="$JAVA_OPTS -XX:+ZGenerational"
JAVA_OPTS="$JAVA_OPTS -XX:SoftRefLRUPolicyMSPerMB=50"
JAVA_OPTS="$JAVA_OPTS -XX:ZCollectionInterval=0"
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:file=/var/log/tomcat/gc.log:time,uptime,level,tags:filecount=10,filesize=100m"
JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
JAVA_OPTS="$JAVA_OPTS -XX:HeapDumpPath=/var/log/tomcat/heapdump.hprof"
export JAVA_OPTS

High‑Throughput (Parallel GC) Configuration

#!/bin/bash
JAVA_OPTS="-Xms8g -Xmx8g"
JAVA_OPTS="$JAVA_OPTS -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m"
JAVA_OPTS="$JAVA_OPTS -XX:+UseParallelGC"
JAVA_OPTS="$JAVA_OPTS -XX:GCTimeRatio=19"   # target GC <5% of total time
JAVA_OPTS="$JAVA_OPTS -XX:MaxGCPauseMillis=500"
JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=8"
JAVA_OPTS="$JAVA_OPTS -XX:NewRatio=2"
export JAVA_OPTS

Real‑World Cases

Case 1: E‑commerce GC Optimization

Problem : During a Double‑11 sale, Full GC pauses of 3‑5 seconds caused timeouts.

Original config : -Xms4g -Xmx4g -XX:+UseG1GC Root causes :

Heap too small for traffic spike.

Object allocation rate exceeded GC capacity.

Many temporary objects promoted to Old Gen.

Solution (increase heap, adjust young‑gen size, trigger Mixed GC earlier, add GC threads):

# Increase heap
JAVA_OPTS="-Xms12g -Xmx12g"
JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"
# Young‑gen size
JAVA_OPTS="$JAVA_OPTS -XX:G1NewSizePercent=40"
JAVA_OPTS="$JAVA_OPTS -XX:G1MaxNewSizePercent=50"
# Earlier Mixed GC
JAVA_OPTS="$JAVA_OPTS -XX:InitiatingHeapOccupancyPercent=35"
# More GC threads
JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=16"
JAVA_OPTS="$JAVA_OPTS -XX:ConcGCThreads=4"

Result after 1 hour monitoring :

Metric                Before   After
Young GC frequency   2×/s     0.5×/s
Young GC pause       ~10 ms   ~15 ms
Full GC frequency    1/min    0 (observed)
P99 response time    5000 ms  200 ms

Case 2: Memory‑Leak Diagnosis

Problem : Application memory grew continuously and eventually OOM.

Investigation steps :

# Enable GC logs and monitor
jstat -gc $(pgrep -f myapp) 1000 10
# Dump heap for analysis
jmap -dump:format=b,file=heap.hprof $(pgrep -f myapp)
# Analyze with MAT or VisualVM
# Live analysis with Arthas
java -jar arthas-boot.jar
# In Arthas console
dashboard
heapdump /tmp/dump.hprof
sc -d *Session*
watch org.apache.catalina.session.StandardSession invalidate '{params,returnObj,throwExp}' -x 2

Root causes :

Session timeout too long (default 30 min).

Sessions not explicitly invalidated on logout.

Reference chains prevented session GC.

Fix (shorten session timeout and invalidate on logout):

<session-config>
    <session-timeout>15</session-timeout>
</session-config>

@PostMapping("/logout")
public void logout(HttpServletRequest request) {
    HttpSession session = request.getSession(false);
    if (session != null) {
        session.invalidate();
    }
}

Case 3: Metaspace OOM

Problem : java.lang.OutOfMemoryError: Metaspace Analysis :

# Check Metaspace usage
jstat -gcmetacapacity $(pgrep -f myapp)
# List loaded classes with Arthas
classloader -l
classloader -t
sc -d *

Root causes :

Heavy use of dynamic proxies, CGLib, reflection generated many classes.

Class‑loader leaks in hot‑deployment scenarios.

Third‑party frameworks repeatedly loading classes.

Resolution (increase Metaspace and enable class unloading):

JAVA_OPTS="$JAVA_OPTS -XX:MetaspaceSize=512m"
JAVA_OPTS="$JAVA_OPTS -XX:MaxMetaspaceSize=1g"
JAVA_OPTS="$JAVA_OPTS -XX:+ClassUnloadingWithConcurrentMark"

Best Practices

Heap Sizing Principles

Set -Xms and -Xmx equal.

Do not exceed ~70 % of physical memory.

Reserve space for Metaspace, Direct Memory, and thread stacks.

# Example for a 32 GB server
Heap: 20 GB
Metaspace: 512 MB
Direct Memory: 2 GB
System & others: ~10 GB

GC Selection Flowchart

Start → Check heap size
  <4 GB → Use G1 (default)
  ≥4 GB → Is latency critical?
    Yes → Use ZGC
    No → Is throughput priority?
      Yes → Use Parallel GC
      No → Tune G1

Monitoring & Alerts

Old‑gen usage > 80 % for 5 minutes.

Full GC occurrence (or business‑defined threshold).

GC time > 5 % of total runtime.

Metaspace usage > 90 %.

Fault Diagnosis Commands

# List Java processes
jps -lv
# GC statistics
jstat -gc PID 1000 10
jstat -gcutil PID 1000 10
# Heap object histogram
jmap -histo PID | head -30
# Export heap dump (use cautiously)
# jmap -dump:format=b,file=heap.hprof PID
# Thread dump
jstack PID > thread.txt
# JVM flags
jcmd PID VM.flags

GC Log Analysis (JDK 21 format)

[2025-01-07T10:30:15.123+0800][12.345s][info][gc] GC(100) Pause Young (Normal) (G1 Evacuation Pause) 1024M->256M(4096M) 15.678ms
[2025-01-07T10:30:15.123+0800][12.345s][info][gc,heap] GC(100) Eden regions: 200->0(180)
[2025-01-07T10:30:15.123+0800][12.345s][info][gc,heap] GC(100) Survivor regions: 20->25(30)
[2025-01-07T10:30:15.123+0800][12.345s][info][gc,heap] GC(100) Old regions: 50->55

Key fields: Pause Young: young‑gen GC. 1024M->256M(4096M): before/after heap usage and total size. 15.678ms: pause duration.

Tools: GCViewer, GCEasy (online).

Arthas for Live Diagnosis

# Start Arthas
java -jar arthas-boot.jar
# Dashboard
dashboard
# Thread view (top 3 busy threads)
thread -n 3
# JVM info
jvm
# Memory info
memory
# Export heap dump
heapdump /tmp/dump.hprof
# CPU profiling
profiler start
... workload ...
profiler stop

Performance Monitoring (Prometheus + Grafana)

Expose JVM metrics with JMX Exporter

# Download JMX Exporter
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.20.0/jmx_prometheus_javaagent-0.20.0.jar
# Minimal config (jmx_exporter.yaml)
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
- pattern: ".*"
# Add to setenv.sh
JAVA_OPTS="$JAVA_OPTS -javaagent:/opt/tomcat/lib/jmx_prometheus_javaagent-0.20.0.jar=9404:/opt/tomcat/conf/jmx_exporter.yaml"

Prometheus scrape config (excerpt)

scrape_configs:
- job_name: 'tomcat'
  static_configs:
  - targets: ['tomcat-server:9404']

Key Metrics (PromQL)

Heap usage:

jvm_memory_bytes_used{area="heap"} / jvm_memory_bytes_max{area="heap"}

GC time ratio: rate(jvm_gc_collection_seconds_sum[5m]) GC count: rate(jvm_gc_collection_seconds_count[5m]) Thread count:

jvm_threads_current

Alert Rules (excerpt)

groups:
- name: jvm_alerts
  rules:
  - alert: JVMHeapUsageHigh
    expr: jvm_memory_bytes_used{area="heap"} / jvm_memory_bytes_max{area="heap"} > 0.8
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "JVM heap memory usage high"
      description: "{{ $labels.instance }} heap usage {{ $value | humanizePercentage }}"
  - alert: JVMFullGC
    expr: increase(jvm_gc_collection_seconds_count{gc="G1 Old Generation"}[5m]) > 0
    labels:
      severity: critical
    annotations:
      summary: "Full GC occurred"
      description: "{{ $labels.instance }} Full GC detected"

Backup & Recovery

Periodic JVM Snapshot Script

#!/bin/bash
PID=$(pgrep -f tomcat)
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/data/backup/jvm"
mkdir -p $BACKUP_DIR
# Thread dump (non‑intrusive)
jstack $PID > $BACKUP_DIR/thread_$DATE.txt
# Copy GC log
cp /var/log/tomcat/gc.log $BACKUP_DIR/gc_$DATE.log
# Keep last 7 days
find $BACKUP_DIR -mtime +7 -delete

OOM Automatic Handling

# In setenv.sh
JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
JAVA_OPTS="$JAVA_OPTS -XX:HeapDumpPath=/var/log/tomcat/heapdump.hprof"
JAVA_OPTS="$JAVA_OPTS -XX:OnOutOfMemoryError='/opt/scripts/oom_handler.sh %p'"

# /opt/scripts/oom_handler.sh
#!/bin/bash
PID=$1
DATE=$(date +%Y%m%d_%H%M%S)
echo "[$DATE] OOM detected for PID: $PID" >> /var/log/tomcat/oom.log
jstack $PID > /var/log/tomcat/oom_thread_$DATE.txt 2>/dev/null
# Optional restart (depends on policy)
# systemctl restart tomcat

Key Takeaways

Understand JVM memory regions (young, old, metaspace, direct).

Choose the right GC collector: G1 for most, ZGC for ultra‑low latency, Parallel for batch‑oriented workloads.

Configure heap size, GC targets, and thread counts based on hardware and workload.

Continuously monitor GC logs, Prometheus metrics, and perform periodic reviews.

Master tools like jstat, jmap, jstack, and Arthas for rapid issue isolation.

Evaluation Criteria

Young GC frequency moderate; pause < 50 ms.

Full GC avoided or pause < 500 ms.

GC time < 5 % of total runtime.

P99 response time meets business SLA.

Further Learning

Deep dive into GC algorithms (tri‑color marking, SATB, Remember Set).

JVM internals (JIT, escape analysis, lock optimizations).

Performance engineering methodology.

Cloud‑native Java (GraalVM, native images, container tuning).

References

Oracle JDK 21 documentation: https://docs.oracle.com/en/java/javase/21/

G1 GC tuning guide: https://docs.oracle.com/en/java/javase/21/gctuning/

ZGC official docs: https://wiki.openjdk.org/display/zgc

Arthas user guide: https://arthas.aliyun.com/doc/