Java CPU 100% Spike: Complete Production Troubleshooting & Root Cause Analysis

Previous six articles covered troubleshooting mindset, logging system, JDK commands, Arthas diagnostics, GC tuning, and server-level commands.

This article begins the real production fatal incident practical module. Instead of learning tools in isolation, we use a complete toolchain to close the loop on the most frequent, interview-essential, and most deadly real-world incidents.

The number one production incident is Java service CPU 100% spike. Almost all senior developers, operators, and architects have handled it. Symptoms: server CPU instantly maxed out, service response timeouts, widespread interface avalanche, alerts flooding, temporary recovery after restart, then recurrence.

Many teams only restart services blindly, treating symptoms not root cause. The core reason: they only know emergency response, not evidence collection and code root cause localization.

This article guides you from zero to reproduce CPU spike, walk through enterprise standard troubleshooting process, analyze thread stacks line by line, locate bug code, provide solutions and long-term post-mortem standards, giving you independent capability to handle CPU avalanche incidents.

1. Real Production Incident Scenario Reproduction

First, simulate a real internet company production incident scenario, fully aligned with daily work:

Incident Symptoms

Sudden massive interface timeout alerts during business peak.

Server CPU spikes to 95%-100%, load stays high.

Service does not crash, no errors, no ERROR stacks in logs, but all interfaces hang with extremely slow response.

Temporary restart restores normal, but issue recurs after a period.

Novice approach: Restart service, gloss over, afraid to dig deeper, next time it explodes again.

Advanced approach: Preserve scene, collect evidence, locate code, root cause fix, post-mortem optimization.

2. Core Root Causes of CPU Spikes (Only 4 Categories in Production)

All Java service CPU 100% spikes, no matter how complex the business, root causes never escape four categories, narrowing your troubleshooting scope:

Business code infinite loops: for/while infinite loops, recursive infinite loops, loops without sleep.

Frequent Full GC / GC explosion: Memory leaks cause GC threads to consume CPU.

Massive lock contention, spin overhead: synchronized, CAS spin crazy retries.

Complex logic time-consuming computation: Large data volume loop traversal, regex backtracking, dense computation logic.

With this scope, troubleshooting is no longer blind guessing; we follow enterprise standard process step by step to collect evidence and locate.

3. Enterprise Standard CPU Spike Troubleshooting Complete Process (Universal Closed-Loop Formula)

This process is the standard SOP for big-tech production incidents; you can save it and execute for all future CPU spikes.

Overall chain: Locate process on server → Locate high-CPU thread → Stack trace forensics → Code localization → Fix and post-mortem

Step 1: top Command to Locate High-CPU Java Process

Run top on server to observe overall resource usage: top Find a Java process with sustained 90%+ CPU, confirm fault process PID. Exclude overall machine load and other processes, confirm business Java process exclusively occupies CPU .

Step 2: top -H to Locate High-Consumption Thread TID Within Process

After determining PID, view all threads' resource usage for that process: top -H -p <process PID> Now you can clearly see: a few business threads continuously max out CPU . Record the super-high CPU thread IDs (TID).

Key distinction:

If business threads : code infinite loops, business logic time-consuming issues.

If GC threads : memory issues, frequent GC causing CPU spike.

Step 3: Convert Thread ID to Hexadecimal (Required for jstack Lookup)

jstack thread IDs are hexadecimal, conversion needed: printf "%x\n" <thread TID> Obtain hexadecimal thread ID for precise stack trace retrieval.

Step 4: jstack Export Thread Stack, Precisely Locate Code Line

Emergency export current thread snapshot on production, preserve fault scene: jstack <process PID> > cpu_thread.log Search stack log by hexadecimal thread ID: grep -A 20 <hex TID> cpu_thread.log Instantly locate: specific class, specific method, specific code line , directly seeing which code segment is stuck in infinite loop consuming CPU.

4. Classic Incident Case Study: Infinite Loop Scenario

We simulate the highest-frequency collection traversal infinite loop real case, fully replicating production bug.

1. Faulty Bug Code

// Error example: list traversal infinite loop
while(true){
    if(list.size() > 0){
        // Business processing
    }
}

Root cause: List non-empty leads to infinite loop, no sleep, no exit condition, thread hogs CPU, instantly saturates core.

2. Stack Trace Characteristics

jstack stack continuously prints the thread in RUNNABLE state, continuously executing current line of code, no blocking, no waiting, infinite execution. This is the typical characteristic of pure code infinite loop .

3. Fix Solution

Add exit condition, add thread sleep, use blocking queue instead of polling:

while(true){
    if(list.size() > 0){
        // Business processing
    } else {
        // Empty queue sleep, release CPU
        Thread.sleep(50);
    }
}

5. Second High-Frequency Scenario: GC Threads Cause CPU 100%

Besides code infinite loops, the second ultra-high-frequency fault: Memory leaks trigger frequent GC, GC threads occupy CPU .

1. Scene Characteristics

top shows CPU maxed out.

top -H reveals highest CPU consumers are GC threads.

jstat -gc observation: FGC continuously surges, GCT extremely high.

Business logs no errors, severe service lag.

Root cause: Memory leak, old generation continuously fills, JVM keeps garbage collecting, CPU exhausted by GC.

2. Troubleshooting Linkage Solution

Combine jmap to view heap memory, analyze large objects and resident objects, locate memory leak code, root fix GC spike issue.

6. Production Emergency Stopgap Measures (First Response When Incident Occurs)

When incident erupts and business suffers, prioritize stopgap then root cause analysis. Standard emergency operations:

Prioritize scene preservation: Do not restart immediately! First jstack export stack, preserve fault evidence.

Temporary business recovery: After evidence export, restart service, or traffic cut to canary, degrade non-core interfaces.

Post-incident deep localization: Based on logs and stacks, locate bug, fix code, deploy patch.

Core taboo: Only restart, no forensics, no post-mortem, incident will inevitably re-explode.

7. Complete Incident Post-Mortem and Long-Term Prevention Standards

Advanced developers troubleshoot not just to solve immediate problem, but to permanently prevent recurrence .

1. Code-Level Prevention

All while/for loops must have termination conditions.

Polling tasks must configure sleep duration to release CPU.

Collection traversal and recursive logic must have boundary checks.

Prohibit empty infinite loops and infinite recursion code from going live.

2. Monitoring-Level Safety Net

Configure CPU usage alert (threshold 80%).

Configure JVM Full GC count alert.

Configure interface latency and throughput monitoring to detect performance risks early.

3. Release Process Safety Net

Code review before release must check loops, recursion, scheduled tasks and other high-risk code, eliminating low-level infinite loop production incidents.

8. Article Summary

This article thoroughly covers the complete troubleshooting closed loop for production CPU 100% spikes : from fault symptoms, root cause classification, enterprise standard SOP, thread forensics, code localization, emergency stopgap, long-term post-mortem — full coverage.

From now on, when facing CPU spikes, you are no longer a novice who only knows restart, but an advanced developer who can forensics, locate, fix, safeguard, and post-mortem , completely mastering the highest-frequency interview and production fault.