Why Is My Java App’s GC Pausing for Seconds? G1 Tuning & MySQL Phantom Reference Fixes

Background

The application experiences GC pauses of up to 4 hours between runs, with request latency exceeding 3 seconds. Investigation revealed an improperly configured G1 garbage collector combined with MySQL connection‑pool idle‑time reclamation that generates many phantom references.

Application Environment

JDK 1.8, mysql‑connector‑java‑5.1.30, commons‑dbcp‑1.4, Spring 4.3.20.RELEASE, hardware: 8 cores 16 GB RAM.

JVM Startup Parameters (original)

-Xms9984m -Xmx9984m -XX:MaxMetaspaceSize=512m -XX:MetaspaceSize=512m -XX:MaxDirectMemorySize=512m -XX:ParallelGCThreads=8 -XX:CICompilerCount=4 -XX:ConcGCThreads=4 -server -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:G1HeapRegionSize=32M -XX:SurvivorRatio=10 -XX:MaxTenuringThreshold=5 -XX:InitiatingHeapOccupancyPercent=45 -XX:G1ReservePercent=20 -XX:G1MixedGCLiveThresholdPercent=80 -XX:MaxGCPauseMillis=100 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+PrintReferenceGC -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -Xloggc:/export/Logs/app/tomcat7-gc.log

DBCP Configuration

maxActive=20
initialSize=10
maxIdle=10
minIdle=5
minEvictableIdleTimeMillis=180000
timeBetweenEvictionRunsMillis=20000
validationQuery=select 1

Problem Investigation

Monitoring showed only young GC and full GC metrics, missing G1 mixed‑GC details. Manual GC logs revealed STW times of 0.10 s for young GC and up to 3.77 s for the remark phase, totaling ~3.87 s.

GC Log Analysis

2023-06-03T14:40:31.391+0800: [GC pause (G1 Evacuation Pause) (young) (initial-mark), 0.1017154 secs]
... (details of parallel workers, reference processing, object copy, etc.)
... [GC remark ... [PhantomReference, 49828 refs, 4.5339260 secs] ...]
... Total STW time = 0.10 + 3.77 = 3.87 secs

The remark phase is dominated by processing phantom references created by the MySQL driver.

MySQL Driver Phantom References

The class com.mysql.jdbc.NonRegisteringDriver stores each connection in a static

ConcurrentHashMap<ConnectionPhantomReference, ConnectionPhantomReference>

. The map holds thousands of phantom references, each retaining a NetworkResources object, causing long reference‑processing times during GC.

public class NonRegisteringDriver implements java.sql.Driver {
    protected static final ConcurrentHashMap<ConnectionPhantomReference, ConnectionPhantomReference> connectionPhantomRefs = new ConcurrentHashMap<>();
    protected static void trackConnection(com.mysql.jdbc.Connection newConn) {
        ConnectionPhantomReference phantomRef = new ConnectionPhantomReference((ConnectionImpl) newConn, refQueue);
        connectionPhantomRefs.put(phantomRef, phantomRef);
    }
}

static class ConnectionPhantomReference extends PhantomReference<ConnectionImpl> {
    private NetworkResources io;
    ConnectionPhantomReference(ConnectionImpl conn, ReferenceQueue<ConnectionImpl> q) {
        super(conn, q);
        try { this.io = conn.getIO().getNetworkResources(); } catch (SQLException e) {}
    }
    void cleanup() { if (io != null) { try { io.forceClose(); } finally { io = null; } } }
}

An AbandonedConnectionCleanupThread daemon continuously removes these references and calls cleanup(), but the sheer volume makes the remark phase expensive.

Solutions

1. Adjust JVM Parameters

Enable parallel reference processing: -XX:+ParallelRefProcEnabled (required for JDK 8).

Set -XX:ParallelGCThreads=8 to match CPU cores.

Lower -XX:G1MixedGCLiveThresholdPercent from 80 to 60 to trigger mixed GC earlier.

Increase -XX:MaxTenuringThreshold from 5 to 15 (or leave default).

Optionally enable dynamic GC threads: -XX:+UseDynamicNumberOfGCThreads.

2. Optimize Connection‑Pool Settings

Increase minEvictableIdleTimeMillis from 180 000 ms (3 min) to a larger value (e.g., 30 min) to reduce premature idle‑connection reclamation.

Reduce minIdle to lower the number of idle connections during low load.

3. Fix MySQL Driver Phantom‑Reference Issue

Upgrade to MySQL Connector/J 8.0.22+ and disable the cleanup thread via system property com.mysql.cj.disableAbandonedConnectionCleanup=true.

Or, for older drivers, periodically clear the static map via reflection:

SCHEDULED_EXECUTOR.scheduleAtFixedRate(() -> {
    try {
        Field f = NonRegisteringDriver.class.getDeclaredField("connectionPhantomRefs");
        f.setAccessible(true);
        Map<?,?> map = (Map<?,?>) f.get(null);
        if (map.size() > 50) map.clear();
    } catch (Exception e) { /* log */ }
}, 2, 2, TimeUnit.HOURS);

Result

After applying the JVM tweaks and fixing the phantom‑reference problem, GC pause times dropped to under 100 ms (average 26 ms) and mixed‑GC frequency decreased dramatically, leaving only young‑GC events.

Summary of G1 Tuning Recommendations

Set ParallelGCThreads to the number of CPU cores.

Set ConcGCThreads to roughly half of the cores.

Understand that -XX:MaxGCPauseMillis is a target, not a guarantee.

Enable -XX:+ParallelRefProcEnabled on JDK 8.

Avoid full GC; focus on young and mixed GC.

MySQL Phantom‑Reference Takeaways

Connector/J < 8.0.22 requires manual cleanup of phantom references.

Connector/J ≥ 8.0.22 can disable the feature via a system property.

These adjustments together resolve the long‑duration GC pauses and improve overall application stability.