Why G1’s Humongous Objects Trigger Midnight Old‑Gen Spikes and How to Diagnose Them

1. Incident

The alarm started at 04:00 when the old‑generation memory of pod‑151 jumped past the 75% threshold; more than 60 pods showed the same pattern between 00:00‑05:00. The service runs with the following JVM parameters (relevant parts only):

-XX:+UseG1GC : enable G1 GC

-XX:G1HeapRegionSize=16 : set G1 region size to 16 MB

-XX:MaxRAMPercentage=75.0 and -X:InitialRAMPercentage=75.0 : configure heap size to about 6 GB

2. Evidence Collection and Reasoning

Two main clues were used: JVM monitoring data (GC count for humongous allocation and old‑gen usage) and heap dumps. The monitoring showed a strong positive correlation between Humongous Allocation and old‑gen memory growth.

Analysis of the heap dump revealed a huge char[] of length 5,636,094 (≈10.8 MB shallow size) that matched the Prometheus exposition format. The array contained many \x00 padding bytes, indicating that the array had been dynamically expanded.

2.1 Humongous Allocation GC

Prometheus metrics are generated by a Spring Actuator endpoint, which writes text to a StringWriter. The writer wraps a StringBuffer that stores characters in a char[]. When the buffer grows, the underlying array expands by roughly newCapacity = oldCapacity*2 + 2, creating humongous objects once the size exceeds half a G1 region (8 MB).

2.2 Gotcha! Big Array!

Using YourKit, the large array was identified and its content confirmed to be Prometheus metrics. The array’s shallow size calculation (Mark Word 8 B + Class Pointer 4 B + Length 4 B + N×2 B + Padding 4 B) matches the 11,272,208 B reported by the heap dump.

3. Suspicion Investigation

Question 1: How is the shallow size of the array computed? – Answered by breaking down the JVM object header and array layout.

Question 2: How do humongous allocation, humongous‑allocation GC, and old‑gen usage interact? – The G1 allocation path attempt_allocation_humongous triggers a GC cycle when old‑gen usage exceeds the Initiating Heap Occupancy Percent (IHOP). The GC performs a Young GC, then a concurrent marking cycle, and finally a cleanup that can reclaim humongous objects (eager reclaim).

Question 3: Why does the alarm only appear at night? – During the day, frequent Young GCs keep the old‑gen growth in check. At night, longer GC intervals allow the old‑gen to accumulate humongous objects, eventually crossing the IHOP threshold and triggering a humongous‑allocation GC.

4. Case Reconstruction

Running a Bash script that repeatedly curls the /actuator/prometheus endpoint reproduces the issue: rapid metric collection creates many humongous arrays, causing old‑gen to grow until a GC is forced.

5. Summary and Recommendations

Key take‑aways:

Beware of dynamic array expansion in classes like StringBuffer, StringBuilder, ArrayList, and I/O streams.

Large primitive arrays become humongous objects and can waste memory due to region alignment and leftover space.

Enable or disable eager reclamation of humongous objects with -XX:G1EagerReclaimHumongousObjects as needed.

When defining large arrays, consider the actual required size and avoid unnecessary capacity parameters.

Monitor GC pause types (young vs. mixed) and IHOP thresholds to understand old‑gen pressure.

Understanding the interplay between G1’s allocation strategy, humongous objects, and GC cycles helps prevent similar memory‑spike incidents.

6. Conclusion

Take care of your apps, don't let them crash!