Big‑Sale Load Test: Why You Shouldn't Rush to Tune JVM Parameters

Don’t Jump to JVM Tuning During a Load Surge

One week before the Double‑11 promotion, the load‑testing platform raised traffic and the order‑creation API’s P99 latency spiked from 90 ms to 700 ms. GC logs showed more frequent Young GC and the appearance of Mixed GC, prompting suggestions to scale the heap, lower MaxGCPauseMillis, or even switch to ZGC—solutions that can easily become traps.

A real‑world case is described where a service changed seven JVM flags the night before a promotion; within 30 minutes Full GC frequency rose from three times per hour to once every ten minutes. The root cause turned out to be a batch interface that loaded 100 000 records into memory at once. Limiting the batch size to 1 000 solved the problem, while none of the JVM parameters helped.

Core Question : How to determine whether the JVM is really responsible for the latency spike?

Build an Evidence Chain Before Adjusting Anything

Enable the necessary diagnostics before the test starts; it’s absurd to finish a test and then realize GC logging was disabled.

JDK 9+ GC logging command :

-Xlog:gc*,safepoint:file=/logs/gc.log:time,uptime,level,tags:filecount=10,filesize=100m

Start a JFR recording :

jcmd <pid> JFR.start name=load-test settings=profile duration=600s filename=/logs/load-test.jfr

JVM metrics alone are insufficient; you need three layers of data:

JVM layer : heap usage, Old Usage After GC, GC pause distribution, allocation rate, promotion rate.

Business layer : interface P99/P999, order success rate, inventory‑lock latency, discount‑calculation time, message‑queue backlog.

Container layer : pod RSS, CPU throttling, I/O wait, container restart count.

This is not a ritual; it is the only way to see the full chain when an incident occurs.

Pre‑warm the JVM before measuring: let JIT finish compiling, classes load, and caches reach normal levels. A cold JVM behaves differently from one that has been running for half an hour.

Pre‑test checklist (ensure each item before the test):

1. Verify startup flags: jcmd <pid> VM.flags
2. Verify heap size: jcmd <pid> GC.heap_info
3. Verify GC log path and rotation size
4. Verify JFR file path has enough space
5. Verify Prometheus/Grafana scrape interval
6. Verify load model matches real‑world traffic mix
7. Verify no batch jobs, exports, or compensation tasks run during the test

After the test, deliver an evidence package instead of a single “GC looks bad, adjust parameters” sentence.

At minimum the package should contain six items:

Test timeline

Core‑interface P99/P999 curves

Overlay of GC pause and business latency

Allocation‑rate and promotion‑rate changes

Old Usage After GC trend

Container RSS and CPU throttling

If possible, add a 10‑minute JFR recording. JFR not only shows GC but also allocation hotspots, lock contention, thread states, and socket read/write times, often surfacing slow SQL, downstream RPC, or object‑allocation hotspots that masquerade as JVM problems.

Focus on the Pivot Point, Not the Final Peak

During the test, look for the “knee” where metrics change dramatically:

At which QPS does Young GC frequency jump from 1 × s⁻¹ to 3 × s⁻¹?

When does promotion rate surge from 10 MB/s to 50 MB/s?

Is Old Usage After GC flat or steadily rising?

Do P99 spikes coincide with GC pauses?

These questions are more valuable than simply noting “1000 QPS can’t be handled.” The request mix matters: a workload of pure product‑detail reads behaves very differently from one that mixes 30 % order creation and discount calculation.

If P99 spikes precede GC pressure, first investigate the business layer—databases, Redis, downstream services, thread‑pool queues—because adjusting the heap will not help.

If GC pauses and P99 spikes align, then proceed with JVM tuning, but keep the order of investigation correct.

Three‑Eye Diagnosis

First eye – Business curve : Does order‑success rate drop? Is the P99 spike isolated to the order interface while other interfaces stay stable?

Second eye – GC curve : Plot GC pause, Young GC frequency, Old Usage After GC together with P99. If P99 spikes first, the business slowdown is likely extending object lifetimes.

Third eye – CPU : If the container’s CPU is already throttled, G1’s concurrent marking threads cannot run, making IHOP adjustments ineffective. Examine CPU throttling, thread‑pool saturation, slow SQL, and RPC timeouts together.

When the following three signals appear simultaneously, fix the thread‑pool, connection‑pool, and timeouts before touching GC settings:

Tomcat threads waiting on locks, connections, or RPC.

Connection‑pool active count at its limit.

P99 spikes occurring after thread‑pool saturation.

These signals indicate that the problem is not a JVM‑parameter issue.

Hard‑Evidence Counter‑Signals

P99 spikes before GC pauses : business threads are slow; objects survive longer, causing later GC pressure.

CPU throttling appears first : container CPU limits starve both application and GC threads.

Connection‑pool saturation first : database, Redis, or HTTP pools hit limits, causing request queuing and higher promotion rates.

When any of these appear, the conclusion should be “JVM parameters are not the highest‑priority issue.”

Typical Missteps

People often assume “it looks like a GC problem, so tweak GC” without evidence, which can hide the real bottleneck and cause failures on the actual promotion day.

Object‑Lifetime and Humongous Regions

If Young GC is too frequent, first examine allocation rate and young‑generation size. For example, allocating 500 MB/s into a 512 MB Eden will trigger Young GC every two seconds. Check whether hot interfaces are creating many temporary objects or whether the load model over‑drives a particular endpoint.

log.info("Order details: " + order.toString()); // concatenates each time
log.info("Order details: {}", order); // at least avoid unnecessary allocation

If Old‑generation growth is rapid, investigate object lifecycles: CompletableFuture closures capturing full order context, ThreadLocal variables not removed, unbounded ConcurrentHashMap caches, large message‑queue backlogs, or batch jobs competing with online traffic for old‑generation space.

When many objects survive, use MAT, JFR Old Object Sample, or jcmd <pid> GC.class_histogram to identify the culprits.

Humongous objects (larger than half a G1 region) often arise from bulk operations such as a multi‑get of 5 000 keys or generating a large CSV report. The remedy is to paginate, batch, stream, or limit the size of each batch rather than simply expanding the heap.

When Full GC or Evacuation Failure Occurs

If the old generation is filled with still‑referenced objects or fragmented by large objects, tuning parameters only masks the symptom. First determine which interface creates the objects, why they live long, and whether G1 needs a larger reclamation window.

Structured Parameter‑Change Process

Document the investigation in a small table with four columns:

Phenomenon : e.g., Young GC frequency, Old Usage trend, Full GC occurrence.

Hypothesis : high allocation rate, long‑lived objects, Humongous fragmentation.

Verification action : examine JFR allocation hotspots, thread stacks, class histogram, container CPU/RSS.

Candidate parameters : only after the above steps.

Changing parameters without this disciplined approach turns tuning into mysticism.

Final Takeaway

During a big promotion, the faster you act, the more you must resist the urge to change JVM flags immediately. Build a solid evidence chain, locate the true bottleneck across business, JVM, and container layers, and only then adjust parameters with a clear hypothesis and verification.