Investigation and Warm‑up Mitigation of JVM C2 CompilerThread CPU Spikes During Service Startup

This article shares a real‑world troubleshooting case from a sorting service where the system experienced request timeouts, high CPU usage, and GC spikes during startup.

Problem Manifestation : At service launch, a wave of timeouts occurred. Monitoring showed large fluctuations in JVM G1 GC, CPU usage, thread‑pool activity, and external I/O latency, accompanied by many system exceptions.

Conclusion : The root cause was the JIT compiler triggering hotspot code compilation with the C2 CompilerThread, leading to high CPU consumption and subsequent request timeouts.

Investigation Process :

1. Initial focus on JVM and GC issues, using gceasy.io and enabling detailed GC logs with the following JVM options:

-XX:+PrintGC 输出GC日志
-XX:+PrintGCDetails 输出GC的详细日志
-XX:+PrintGCTimeStamps 输出GC的时间戳（以基准时间的形式，你启动的时候相当于12点，跟真实时间无关）
-XX:+PrintGCDateStamps 输出GC的时间戳（以日期的形式，如 2013-05-04T21:53:59.234+0800）
-Xloggc:../logs/gc.log 日志文件的输出路径

Various JVM tuning attempts (heap size, GC threads, pause time targets) proved ineffective. A temporary sleep after model loading reduced GC pressure but did not solve the timeout issue.

2. Shifted focus to CPU analysis after observing synchronized spikes in thread‑pool, I/O, and request latency. Used Arthas ( documentation ) and traditional tools (top, jstack) to inspect CPU usage at the moment traffic arrived.

The profiling revealed that the C2 CompilerThread was consuming a large portion of CPU.

Understanding the C2 CompilerThread : Java initially runs bytecode in interpreter mode. Frequently executed code is JIT‑compiled. The JIT has multiple tiers; C2 is the highest‑optimization tier (Tier‑5). While C2 provides the best performance, its compilation phase is CPU‑intensive, which caused the observed spikes.

3. Attempted mitigations:

• Disabled tiered compilation (forcing C1) with -XX:-TieredCompilation -client . This removed the C2 thread but resulted in overall poorer performance and sustained high CPU.

• Increased C2 compiler thread count with -XX:CICompilerCount=8 (and later restored tiered compilation). This yielded modest improvement but did not eliminate timeouts.

4. Final solution – Warm‑up (pre‑heat) strategy:

During normal operation, a portion of live traffic is recorded and serialized to files. On service restart, the recorded requests are replayed before accepting real traffic, allowing the JIT to compile hotspot code ahead of time. This eliminates the CPU spike caused by on‑the‑fly C2 compilation.

Implementation details include:

Using an AOP Around advice with a custom annotation to capture successful request objects without affecting normal flow.

Deep‑copying request objects before serialization to avoid side‑effects.

Storing serialized requests per business code in separate directories to preserve generic types.

Parallelizing the warm‑up replay with multiple threads, reducing warm‑up time from ~12 minutes to ~3 minutes.

Results showed a reduction of ~10,000 exception requests per deployment, smoother release processes, and stable response times (RT) after warm‑up.

References include articles on -XX:+TieredCompilation , C2 CompilerThread analysis, and JVM optimization literature such as "深入理解Java虚拟机".