Don't Blame Redis: Mixed GC Pauses Extend Lock Hold Times in G1
This article explains how frequent G1 Mixed GC pauses, not Redis slowness, cause distributed lock waits by stalling lock-holding threads, and details a six-step troubleshooting process linking GC logs, business logs, and heap analysis to identify large objects, compensation tasks, and unbounded local caches as root causes.
When cache updates slow down, interfaces time out, and distributed locks wait longer, the instinct is to blame Redis. However, if Redis CPU, network, and command latency are normal, the real culprit may be the JVM: the thread holding the lock is repeatedly paused by Mixed GC, preventing it from releasing the lock and forcing subsequent requests to queue.
Lock Waits Are the Starting Point
In a typical product cache sync flow, a message queue triggers a cache service to update Redis and a local cache, guarded by a per-shop distributed lock. During peak promotions, compensation tasks, scheduled syncs, and bulk uploads converge, increasing lock contention. Logs may show lock acquisition timeouts, rising cache write latency, and slower product APIs. While Redis metrics (CPU, network, slow queries, lock contention) should be checked, the critical next step is to examine what the lock-holding application instance was doing at that moment.
Correlate the instance's GC logs with lock acquire/release timestamps. If prolonged lock holding coincides with multiple Mixed GC pauses, there is reason to investigate the JVM further. However, temporal overlap alone is insufficient; you must verify whether pauses occurred during the lock hold, sum their duration, and determine how much of the extra latency they explain. Database queries and remote calls inside the lock can also contribute.
When the Lock-Holding Thread Pauses, Everyone Waits
Mixed GC requires a stop-the-world (STW) pause: it collects young regions plus a subset of old regions. If a cache task holds a shop lock and then fetches product data, builds collections, assembles JSON, writes to Redis, and updates the local cache, any GC pause halts all these steps, delaying lock release. Other requests queue behind the lock, like people waiting outside a storeroom while the key holder is frozen.
If the lock has a lease, a long pause risks lease expiry, raising safety concerns for the resumed task. The more work inside the lock, the longer the base hold time, the higher the chance of intersecting a GC pause. Combined with queuing from the same shop, interface latency can amplify to seconds. Importantly, this is not a single multi-second GC pause, but the sum of business processing, GC pauses, and queueing time.
GC pressure may originate from other threads in the same JVM allocating heavily, not just the lock-holding task. Therefore, investigate both the lock's internal workload and the entire process's allocation and retention patterns.
Why Did Old Generation Grow Despite Infrequent Compensation Tasks?
Compensation tasks often hide risk in "how much data per run." Examples: loading massive product DTO lists, aggregating multi-shop data into large maps, or serializing full JSON in memory before writing. These create two pressure types:
Single huge objects: G1 treats objects larger than half a region as Humongous , allocating them directly in old generation's dedicated regions. A large List with product data may not itself be Humongous, but its backing array, a Map 's bucket array, or a serialization byte[] can be.
Many long-lived objects: Even small objects, if retained for the task's duration or cached locally, survive multiple GC cycles and gradually inflate old generation occupancy.
Humongous allocations may trigger earlier concurrent marking, but do not directly imply more frequent Mixed GC. Humongous objects have their own reclamation path and are not simply part of the old regions collected each Mixed GC. The key is to observe memory evolution after task start: Humongous growth, ordinary old object increase, and how much space each Mixed GC actually reclaims.
Mixed GC selects only a portion of old regions. If those regions contain many live objects, reclamation cost is high and freed space limited. Consecutive Mixed GCs are not alarming per se; the warning sign is increasing pause frequency while old generation usage fails to drop after reclamation.
Thus, risk assessment must go beyond QPS. A low-frequency compensation task can consume more memory than many ordinary queries. Look at how many objects are allocated per run and how long they are retained .
Should IHOP Be Lowered? Check Adaptive IHOP First
A common configuration seen during troubleshooting: -XX:InitiatingHeapOccupancyPercent=30 Before judging whether 30 is high or low, confirm the JDK version and whether adaptive IHOP is enabled. Since JDK 9, G1 enables adaptive IHOP by default. In this mode, InitiatingHeapOccupancyPercent serves only as the initial threshold before G1 gathers enough samples; thereafter, the actual threshold adjusts based on old generation allocation rate, marking duration, etc. Setting a fixed initial value may not solve issues that appear after the adaptive mechanism has taken over.
If adaptive IHOP is disabled ( -XX:-G1UseAdaptiveIHOP), a fixed threshold requires balancing: too high risks starting marking too late, leaving insufficient heap for subsequent collection; too low triggers concurrent cycles prematurely, increasing marking and pause overhead. Adaptive IHOP adjusts based on observed load but cannot perfectly predict sudden bursts. A stable allocation rate may be upended by a large batch compensation task, causing predictions to lag.
Correlate business logs with GC logs: when did the compensation task start? When did old generation and Humongous rise? Did concurrent marking start promptly? How effective were subsequent Mixed GCs? Only if logs and load tests prove the adaptive threshold unsuitable for the current workload should you consider disabling adaptive IHOP and validating a fixed value. Do not disable adaptive IHOP merely because a few GCs were dense.
IHOP controls when marking starts . If massive objects remain strongly referenced by business logic, earlier marking cannot reclaim them.
Before Tuning Parameters, Identify Where Mixed GC Spends Time
Several parameters influence Mixed GC behavior: G1HeapWastePercent – allowed reclaimable space percentage, affects whether continuing Mixed GC is worthwhile. G1MixedGCLiveThresholdPercent – old region live data percentage above which the region is excluded from Mixed candidates. G1MixedGCCountTarget – target number of Mixed GCs to spread candidate old regions across after one marking cycle; actual count may differ. G1OldCSetRegionThresholdPercent – maximum old regions per Mixed GC, as a percentage of total heap regions. ParallelGCThreads – GC worker threads for STW phases; actual count may be dynamically adjusted. ConcGCThreads – GC threads for concurrent phases like marking.
These parameters tune collection scope, per-cycle work, and speed. However, they cannot collect objects still in use by the business, nor can they split oversized compensation tasks. Before adjusting, examine GC logs to see where time goes: object copying, cross-region reference processing (Scan RS, Update RS), single pause length, or insufficient reclamation per cycle leading to sustained old generation growth. Different symptoms demand different tuning directions.
More GC threads are not always better. During STW, business threads are already paused; adding parallel GC threads yields diminishing returns and adds scheduling overhead. Concurrent marking runs alongside business threads; extra threads may contend for CPU needed by request processing. After tuning, evaluate both GC pause times and overall CPU usage and business throughput, not just whether one pause shortened.
Split Batches First, Then Shorten Lock Hold Time
If batch tasks and cache retention are identified as the source, apply these business-side fixes:
Loading too many products at once: paginate, process in batches, use incremental diffs to avoid full loads.
Building large collections inside the lock: move preparatory work outside the lock while preserving data consistency guarantees.
Compensation tasks competing with main pipeline: schedule off-peak, limit concurrency and processing rate, pause compensation when main pipeline is busy.
Large lists/maps held indefinitely: control batch size, dereference unneeded objects promptly after processing.
Unbounded local cache: set capacity, expiration, and eviction policies.
Rising Humongous usage: pinpoint the specific large objects, adjust batching and serialization, then evaluate region size.
When narrowing lock scope, be careful: preparatory data that can be computed beforehand should stay outside the lock; state that must be checked and updated atomically must remain inside. Do not sacrifice correctness of concurrent updates for shorter hold times.
Encode these limits in code and configuration: max products per batch, max local cache size, max concurrent compensation tasks, pause compensation when main pipeline is busy. Explicit ceilings are more reliable than "remember to control batch size."
After changes, re-verify under comparable load: did large object allocation decrease? Did lock hold time and wait time drop? Did old generation usage stabilize after reclamation? GC may still pause the lock holder; the goal is to reduce the business impact of pauses and queuing, not to eliminate their correlation entirely.
Six-Step Troubleshooting Checklist
Pinpoint the slow requests. Identify the shop, the lock, the instance. Distinguish slow lock acquisition, long lock hold, and slow Redis commands.
Align lock hold time with GC pauses. Verify pauses occurred during lock hold, sum pause duration, and quantify how much extra latency they explain.
Observe heap changes before and after collection. Check if old generation usage dropped, if Humongous grew continuously, and correlate with compensation task execution windows.
Find the most time-consuming GC phases. Focus on Object Copy, Scan RS, Update RS (names vary by JDK version; refer to actual logs).
Trace object allocation and retention. Use allocation records or heap dumps to see if large arrays/collections originate from compensation tasks, bulk queries, or local caches; follow reference chains to discover why they cannot be collected.
Return to code to confirm fix points. Verify batch sizes, cache limits, lock scope, and presence of slow remote calls inside the lock; re-validate with the same metrics after changes.
Business logs tell you where it's slow; GC logs tell you how long the pauses were; object and reference analysis explains why memory won't shrink. When all three align, your fixes have evidence.
The Lock Waits, But the Cause May Be in the JVM
When Mixed GC is frequent and Redis lock waits grow, resist the urge to assign blame immediately or tweak a parameter at first sight. First, confirm whether the lock-holding instance experienced pauses. Then, identify who is creating massive objects and which references are never released. Lock internal workload, compensation batch size, and cache capacity must be examined together.
Lock waits are not necessarily due to a slow Redis; the lock holder may be paused by the JVM. Clarify that timeline before deciding whether to fix the lock, the task, or the GC.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
