JVM Crash Caused by JIT Compilation in a Scheduled Task System and How to Resolve It
A scheduled task system that frequently accesses MySQL and cache crashes at night due to a SIGSEGV in the HotSpot JVM caused by JIT compilation of hotspot code, and the article explains the JVM internals, JIT behavior, and provides configuration and driver downgrade solutions to prevent the failure.
The system is a scheduled‑task service that periodically runs business code accessing MySQL and a cache. After running normally for a while, the Java process disappears at midnight, leaving only a crash log indicating a fatal SIGSEGV in the HotSpot JVM.
cat: /proc/1/environ: Permission denied [admin@host-11-40-38-52 ~]$ more hs_err_pid231.log # A fatal error has been detected by the Java Runtime Environment: # SIGSEGV (0xb) at pc=0x00007f21a8c21325, pid=231, tid=139779725313792 # JRE version: Java(TM) SE Runtime Environment (8.0_20-b26) (build 1.8.0_20-b26) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.20-b23 mixed mode linux-amd64 compressed oops) # Problematic frame: # V [libjvm.so+0x858325] LoadKlassNode::make(PhaseGVN&, Node*, Node*, TypePtr const*, TypeKlassPtr const*)+0x45 # Core dump written. Default location: /home/admin/core or core.231 # ... (full crash log omitted for brevity) ...
After the JVM finishes initialization, bytecode is first interpreted; when a method becomes hot, the Just‑In‑Time (JIT) compiler translates it to native machine code for better performance. HotSpot uses two JIT compilers: the fast, locality‑focused C1 (Client) compiler and the optimizing C2 (Server) compiler.
Hotspot detection relies on two counters per method: an invocation counter and a back‑edge counter. When the sum exceeds a threshold (default 1500 for C1, 10000 for C2, configurable via -XX:CompileThreshold ), the method is considered hot and JIT compilation is triggered.
In this case, the scheduled task repeatedly executed the same code, causing the JIT compiler to compile it. During compilation an internal error occurred, leading to the JVM crash.
Two main remediation approaches are suggested:
Adjust JVM options to control JIT behavior, for example: java -Xint -version # interpreter‑only mode java -Xcomp -version # compile‑only mode java -Xmixed -version # mixed mode (default) -XX:+PrintCompilation # print JIT compilation info -XX:CompileCommand=exclude,package/Class,methodName # exclude specific method from JIT -XX:CompileCommand='compileonly,java/lang/StringBuffer.*' # compile only specific methods -XX:CompileThreshold=1000 # lower the compilation trigger threshold
Rollback the MySQL driver to a stable version, as the issue appeared after upgrading to the driver bundled with spring-boot-2.1.6.RELEASE . Using an older, proven driver eliminates the crash.
By either limiting JIT compilation for the problematic method or using a stable JDBC driver, the scheduled task can run continuously without causing the JVM to abort.
JD Retail Technology
Official platform of JD Retail Technology, delivering insightful R&D news and a deep look into the lives and work of technologists.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.