Hidden GC Pauses Causing Random API Timeouts: JVM STW Troubleshooting Guide

Real Production Fault Scenario Reproduction

The article begins by simulating a typical hidden production fault in internet companies:

Service runs stably overall — no large-scale timeouts, no service crashes, no ERROR logs

Interfaces respond normally (20–50 ms) most of the time, but randomly and occasionally spike to 1–3 second timeouts

Monitoring panels show severe spikes in interface latency curves

Investigation of business code, database, third-party calls, and thread pools reveals no anomalies

Issue has no fixed pattern; reproduces randomly during both peak and off-peak hours

Common misdiagnosis: Developers repeatedly check code, slow SQL, network, and gateway rate limiting — finding nothing.

Actual root cause: JVM GC single STW (Stop-The-World) pause too long; all business threads freeze globally, causing interface requests to hang and timeout.

Core Principle: Why GC Pauses Cause Interface Timeouts

Key concept: STW (Stop-The-World) . During garbage collection (especially marking and compaction phases), the JVM pauses all user business threads , leaving only GC threads running. This pause is the STW global pause.

Normal Minor GC: pause extremely short (a few milliseconds), imperceptible to users, no business impact

Abnormal long-pause GC: single STW duration reaches hundreds of milliseconds, 1 second, or even several seconds

Interface request happens to hit the STW pause window: business thread freezes, cannot execute, triggers interface timeout

This explains why code has no bugs, resources have no bottlenecks, logs show no errors, yet interfaces occasionally timeout .

Two High-Frequency Fault Scenarios (Full Coverage of Production Issues)

Scenario 1: Minor GC Single Pause Too Long (Most Hidden, Highest Frequency)

Many developers assume Minor GC is lightweight and won't affect business — a major misconception. When young generation memory allocation is unreasonable, object promotion is abnormal, or memory fragmentation is severe, ordinary young generation Minor GC can also exhibit hundreds of milliseconds or even second-level STW pauses, triggering random interface timeouts.

Scenario 2: Old Generation GC / Full GC Long Pause

Insufficient old generation space, memory leaks, metaspace pressure, or heap memory fragmentation trigger old generation GC or Full GC. These GCs inherently have longer STW times and easily breach interface timeout thresholds (default 1s/3s), causing interface timeouts.

Enterprise Standard Troubleshooting SOP (Precise Location of GC Long Pauses)

Such hidden faults cannot be located by eye-balling logs or monitoring curves; they require precise GC log analysis . Below is a reusable standard troubleshooting flow.

Step 1: Enable Detailed GC Logs (Production Mandatory)

To troubleshoot GC pause issues, complete GC log parameters must be deployed, retaining every GC's duration, type, and pause time. Production JVM essential configuration:

# Print detailed GC logs
-XX:+PrintGCDetails
# Print GC timestamps
-XX:+PrintGCDateStamps
# Output GC log file
-Xloggc:/data/logs/gc.log
# Print detailed STW pause time
-XX:+PrintGCApplicationStoppedTime

Core parameter PrintGCApplicationStoppedTime : precisely records every global pause duration — the key evidence for troubleshooting long pauses.

Step 2: Filter Long-Pause GC Records

Focus on the key field in GC logs: Total time for which application threads were stopped

Normal business GC pause: 0.005s, 0.01s (millisecond level)

Fault abnormal pause: 0.8s, 1.2s, 2.5s (hundreds of milliseconds / second level, directly triggers timeout)

As long as this pause time exceeds the interface timeout threshold, 100% will produce random interface timeouts and latency spikes .

Step 3: Distinguish Long-Pause Root Cause Types

Combined with GC log type, quickly determine the problem source:

Young generation GC long pause : young generation memory too small, dynamic object age promotion, frequent GC, memory fragmentation

Old generation GC long pause : old generation memory insufficient, too many large objects, memory leak, heap compaction time-consuming

Full GC long pause : metaspace overflow, manual System.gc() trigger, overall heap space pressure

Four Core Root Causes of GC Long Pauses (Source of 99% Production Issues)

1. Young Generation Memory Allocation Too Small

Young generation space insufficient, objects frequently trigger Minor GC, and each collection must scan large numbers of surviving objects, copying overhead huge, causing single GC pause to spike. Typical characteristics: extremely high GC frequency, each pause duration gradually lengthens, frequent interface jitter.

2. Instantaneous Large Objects / Large Collections Online

Business instantaneously queries massive data, creates super-large List/Map in one go, parses super-large JSON files, generating large numbers of temporary large objects. Large objects cannot survive in young generation, directly enter old generation, trigger old generation GC, STW time greatly extended, instantly breaching timeout threshold.

3. Severe Heap Memory Fragmentation

Long-term frequent object creation and destruction leads to heap memory fragmentation; free space scattered and non-contiguous. JVM GC organizing memory and compressing space consumes massive time, causing ultra-long STW pauses. Typical characteristic: service runs longer, GC pauses get longer; restart instantly restores performance.

4. Unreasonable GC Collector Selection

Legacy services use Parallel GC, CMS collector; under high concurrency, large memory scenarios, marking, compaction, fragmentation cleanup efficiency extremely low, prone to long pauses. Especially CMS concurrent failure, floating garbage issues trigger fallback Full GC, causing second-level pauses.

Landing Optimization Solutions (Short-Term Stopgap + Long-Term Cure)

1. Emergency Stopgap Solution (Quickly Eliminate Timeout Spikes)

Temporarily restart service, clear memory fragmentation, restore initial GC performance

Temporarily increase young generation memory, reduce GC trigger frequency

Prohibit business instantaneous full queries, loading super-large collections, avoid large object generation

2. Long-Term Cure Optimization Solution (Production Standard Landing)

Reasonable heap memory ratio allocation : optimize young generation, old generation memory proportion, avoid young generation too small causing frequent GC

Govern business large objects : all batch queries forced pagination, large file shard parsing, eliminate one-time full data loading

Replace with high-performance GC collector : high-concurrency online services upgrade to G1/ZGC, drastically reduce STW pause time; ZGC achieves millisecond-level ultra-low pause

Disable manual GC : enable parameter -XX:+DisableExplicitGC, prohibit code manually calling System.gc() triggering useless GC

Optimize memory fragmentation : G1 collector enables automatic memory compaction, periodically compress heap space, reduce fragmentation issues

Monitoring Safety Net: Early Warning of GC Long Pause Risks

The scariest aspect of such hidden faults is no prior alert; only after users perceive lag and timeout is the problem discovered. Production must configure the following monitoring metrics to intercept risks early:

Single GC maximum pause time alert (threshold 200ms, alert if exceeded)

GC average pause duration trend monitoring

Interface latency P95/P99 monitoring (precisely capture instantaneous timeout spikes)

Old generation memory usage trend monitoring, early detect memory fragmentation, leak issues

Article Summary & Interview Review

This article solves the most hidden, most easily overlooked online fault in the JVM system: GC long pause causing random interface timeouts.

Core key points summary:

Interface no error, occasional timeout, latency spikes — prioritize investigating JVM GC STW pause time

Not only Full GC affects business; abnormal Minor GC long pause equally triggers faults

Large objects, memory fragmentation, unreasonable memory ratio, outdated GC selection are four core root causes

Optimization core: reduce GC frequency, shorten STW pause, eliminate useless GC, govern business large objects

Interview bonus talking point : For online occasional no-error timeouts, I prioritize checking STW pause time in GC logs, distinguish whether young generation GC or old generation GC causes long pause, solve GC pause-induced business jitter by optimizing memory ratio, governing large objects, upgrading GC collector.

Next Episode Preview

Next article enters Episode 13: JVM Parameter Tuning Practice — Production Optimal Configuration, Different Server Configuration Adaptation Solutions , integrating all previous GC, memory, fault knowledge points, delivering a set of universally optimal JVM tuning parameters directly copyable to production, adapting low/mid/high spec servers, completely bidding farewell to random parameter configuration and blind tuning.