CPU 100% at 3 AM: Complete Troubleshooting Guide with Real Incident Analysis

Problem Background

At 3 AM an alert fires: one web server in the cluster shows 100% CPU for 15 minutes, homepage loads slowly, and some APIs time out. SSH into the machine and top reveals an unknown Java process consuming 800%+ CPU on a multi-core box, while other processes queue up.

This is not an isolated CPU spike but a typical fault requiring a full investigation chain. The article uses this real incident to systematically explain the troubleshooting mindset, core commands, investigation paths, fixes, and postmortem methods for Linux CPU 100% issues.

Core Concepts

CPU Usage Breakdown

On Linux, CPU time splits into:

User (us) : application code and library calls. High us means the application's own compute logic is the bottleneck.

System (sy) : kernel work via syscalls (I/O, memory allocation, scheduling). High sy indicates heavy syscall activity or scheduler contention.

I/O wait (wa) : CPU waiting for disk or network I/O. High wa points to storage or network bottlenecks.

Hard/soft interrupts (hi/si) : hardware and software interrupt handling. Spikes may signal NIC packet storms or storage interrupt floods.

Idle (id) : free CPU. id=100% means fully idle. top shows a snapshot; watch trends over time, not a single sample.

Multi-core Numbers

On an 8-core machine a process showing 800% CPU simply uses 8 cores fully (800%/100% = 8 cores). top reports per-core percentages, so >100% is normal. Judge true saturation by checking if idle (id) is near 0 and whether load average far exceeds core count. Load average (1/5/15 min) of 8 on 8 cores means just saturated; 16 means 8 processes queued.

Java CPU Accounting

A JVM process consumes CPU from three sources:

Business code execution (algorithms, JSON serialization, etc.)

JVM internals: GC, JIT compilation, class loading, interpreter overhead

JNI calls: native library CPU time accounted in the Java process's user time

Troubleshooting Path (7 Steps)

Step 1: Confirm Scope & Impact

Before diving, establish the blast radius: single host or cluster-wide? Business impact magnitude? When did it start?

# Check all hosts (Ansible/SSH loop)
for host in $(cat /tmp/hosts.txt); do ssh $host "uptime; top -bn1 | head -5"; done

# Kubernetes
kubectl top nodes
kubectl get pods -o wide | grep -v Running

Confirm duration and scope to set urgency and rollback window.

Step 2: Identify the Offending Process

SSH to the bad host, run top, press Shift+P to sort by CPU. Note PID and command. Press c for full command line, H for thread view.

If the process name looks random, suspect compromise. Check its executable path and network connections:

ls -la /proc/<PID>/exe
cat /proc/<PID>/cmdline | xargs -0 echo
ss -tunapl | grep <PID>
netstat -anp | grep <PID>
lsof -p <PID>

Step 3: Analyze CPU Source (User vs Kernel)

top -bn1 -p <PID>
pidstat -p <PID> 1 5

pidstat

shows %usr, %system, %guest, %CPU. High %usr → application compute; high %system → syscall storm.

Step 4: Java-Specific Deep Dive

GC Analysis with jstat

jstat -gcutil <PID> 1000 10

Key columns:

S0/S1 : Survivor usage. Sustained >80% → rapid object aging.

O : Old gen usage. Rising O triggers Full GC (STW).

YGC/YGCT : Young GC count & time. Rapid YGC rise → high allocation rate.

FGC/FGCT : Full GC count & time. High FGC → Old gen repeatedly filling.

GCT : Total GC time. GCT / uptime >30% → GC is the main bottleneck.

If Full GC is frequent and GCT high, CPU spike is GC-driven. Next, examine GC logs.

Enable & Read GC Logs

If missing, add JVM flags (requires rolling restart):

-Xlog:gc*:file=/var/log/myapp-gc.log:time,uptime,level,tags:filecount=10,filesize=10M
# Java 8 example
-Xloggc:/var/log/myapp-gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M

Analyze with grep/awk:

grep "Full GC" /var/log/myapp-gc.log | wc -l
grep "Full GC" /var/log/myapp-gc.log | awk '{sum += $NF} END {print sum}'
grep "Full GC" /var/log/myapp-gc.log | tail -1

Thread Analysis with jstack

jstack <PID> > /tmp/jstack-<PID>-$(date +%Y%m%d%H%M%S).log

Focus on:

Deadlock section – highest priority.

Runnable threads – if many runnable and CPU high, they're doing compute-heavy work. Correlate top thread IDs (decimal) to jstack nid (hex): printf '%x\n' <decimal_tid> then grep -A 20 "nid=0x<hex>".

Blocked – lock contention.

Waiting on condition – I/O, lock, condition variable waits.

Arthas (Recommended)

curl -O https://arthas.aliyun.com/arthas-boot.jar
java -jar arthas-boot.jar
# Inside Arthas:
dashboard          # threads & CPU
thread -n 10       # top 10 CPU threads with code location
thread             # all thread states
jvm                # memory, GC, class loading

Arthas thread -n directly shows CPU-hot threads and their stack, easier than manual top + jstack correlation.

Step 5: Other Language Processes

Python

pidstat -t -p <PID> 1 5
lsof -p <PID>
python -m pip install py-spy
py-spy record -o /tmp/profile.svg --pid <PID>

Python's GIL limits true parallelism; multi-core 100% often means multiple single-core Python workers.

Go

# Requires net/http/pprof imported
curl http://localhost:6060/debug/pprof/profile?seconds=30 > /tmp/cpu.prof
go tool pprof /tmp/cpu.prof
# Inside pprof: top, web

Go's runtime CPU profile is more precise than Java's jstack.

Step 6: Syscall Tracing

# Syscall stats (heavy overhead – use carefully)
strace -cp <PID> -f
# Specific syscalls
strace -e trace=read,write,epoll_wait -p <PID>
# Perf sampling (lower overhead)
perf record -F 99 -p <PID> -g -- sleep 30
perf report

strace

can slow a process 10x+; prefer -c stats mode. perf needs install and permissions.

Step 7: Root Cause & Fix

Typical root causes and remedies:

Frequent GC : tune heap ( -Xmx), switch GC (G1→ZGC), reduce allocation hotspots (loop allocations, string concat, JSON). Temp: increase -Xmx. If Old gen leaks, use MAT on heap dump.

Infinite loop / compute hotspot : fix logic (missing break), or scale out / optimize algorithm if legitimate.

Lock contention : from jstack find longest-held lock; reduce granularity (sharding), use ConcurrentHashMap, ReentrantReadWriteLock.

Downstream timeout → thread pile-up : check pool config (DB/HTTP). If MySQL/Redis/HTTP slow, threads block in I/O; CPU may not be high but business stalls. Verify with ss -s or netstat -an | grep TIME_WAIT.

Extended Analysis: Deeper Performance Tools

Flame Graphs

Visualize CPU hotspots. Generate with perf + Brendan Gregg's FlameGraph:

perf record -F 99 -a -g -- sleep 30
git clone https://github.com/brendangregg/FlameGraph.git
perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > cpu.svg

Interpretation: x-axis = CPU share, y-axis = stack depth. Wide flat-topped blocks = hot functions.

Java Heap Analysis with MAT

If memory pressure (frequent OOM/GC), dump heap:

jmap -dump:format=b,file=/tmp/heap-$(date +%Y%m%d%H%M%S).hprof <PID>
# Or Arthas (lower overhead)
heapdump /tmp/heap.hprof

Open in Eclipse MAT. Key views: Leak Suspects, Dominator Tree, Top Consumers, Histogram. Typical leak signatures: unreleased char[] / byte[], ever-growing collections ( HashMap, ArrayList), accumulating ThreadLocal objects.

System-Level CPU Analysis

vmstat 1 5

– r (runnable queue) > cores = queuing; us/sy/id/wa/st breakdown. mpstat -P ALL 1 5 – per-core usage; a single hot core suggests single-threaded bottleneck. iostat -x 1 5 – %util near 100% = disk bottleneck. High %util with low wa = sequential I/O; high both = random I/O or full disk.

Network I/O & CPU

High concurrency raises softirq ( NET_RX / NET_TX). Check:

cat /proc/softirqs
cat /proc/interrupts | grep -i "eth\|network"
ethtool -g eth0   # ring buffer
ethtool -l eth0   # multi-queue config

Cache & Memory Bandwidth

perf stat -e 'cpu-clock,cpu-migrations,context-switches,page-faults,cycles,instructions,branches,branch-misses,cache-references,cache-misses' -p <PID> -- sleep 10

High cache-misses → poor data locality; optimize memory layout for spatial locality.

Case Study: The Midnight JVM Memory Avalanche

Incident Timeline

03:15 – Alert: Order Service host A CPU 98%, P99 latency 200ms → 8s, timeouts surge.

SSH + top: Java process 900%+ on 16-core box. jstat (10 samples): Old gen 99%, Full GC every 30s, each 5-8s STW → business threads paused, request backlog grows, GC threads burn CPU.

JVM flags: -Xmx4g -Xms4g – 4G heap insufficient for current load.

Arthas dashboard: 200 threads in order-service pool, 180 in TIMED_WAITING (DB wait).

DBA confirms: replica DB locked by backup job → replication lag → replica queries timeout.

Causal chain: MySQL backup locks replica → replica queries timeout → order service threads block → unreleased request objects fill Old gen → frequent Full GC → CPU 100%.

Fixes

Immediate (10 min):

Kill backup job on replica → locks released, queries recover.

Watch GC: Old gen drops, Full GC frequency falls.

Business recovers, CPU normalizes.

Permanent (3 days):

Move backup to off-peak; add --single-transaction for lock-free InnoDB backup.

Raise JVM heap -Xmx 4G → 8G for burst headroom.

Add replica lag monitoring (>30s alerts).

Implement circuit breaker in order service: failover to primary when replica lag exceeds threshold.

Lessons Learned

CPU 100% is a symptom, not the root cause. The true root was a DB backup job; CPU spike was the final cascade layer. Troubleshooting must go bottom-up: infrastructure → middleware → application. Staring at CPU metrics alone leads nowhere.

Postmortem Checklist

☐ Incident start time
☐ Detection time
☐ Resolution time
☐ Total impact duration
☐ Affected business/users
☐ Root cause
☐ Trigger condition
☐ Amplifying factors
☐ Temporary mitigation
☐ Permanent fix
☐ Monitoring gaps (add new alerts)
☐ Process improvements
☐ Prevention measures

Command Cheatsheets

Basic Monitoring

top                 # system & process CPU, Shift+P sort
htop                # visual top, tree view
uptime              # load average
mpstat -P ALL 1 5   # per-core usage
vmstat 1 10         # CPU, mem, swap, I/O
iostat -x 1 5       # disk I/O detail
sar -u 1 5          # CPU stats

Process Analysis

top -bn1 -p <PID>           # single sample
pidstat -p <PID> 1 10       # continuous stats
top -bn1 -H -p <PID>        # thread-level CPU
strace -cp <PID> -f         # syscall stats (careful)
ltrace -p <PID>             # library calls

Java Process Analysis

jstat -gcutil <PID> 1000 10   # GC stats 1s x10
jstat -gccapacity <PID>       # GC capacity details
jstack <PID> > /tmp/jstack.log
jinfo -flags <PID>            # JVM flags
jmap -heap <PID>              # heap usage
jmap -dump:format=b,file=/tmp/heap.hprof <PID>  # heap dump

Flame Graph & Profiling

perf record -F 99 -a -g -- sleep 30
perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > cpu.svg
# Java Flight Recorder
java -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:StartFlightRecording=duration=30s,filename=/tmp/recording.jfr -jar app.jar

Network & Memory

ss -tunapl | grep <PID>
netstat -an | grep <PID>
lsof -i :8080
free -m
pmap -x <PID>

System Internals

pstree -p <PID>                    # process tree
lsof -p <PID>                      # open files
cat /proc/<PID>/environ | tr '\0' '
'
cat /proc/<PID>/cmdline | xargs -0 echo

Configuration Examples

JVM Memory (8-core 16G)

java -Xms8g -Xmx8g \
  -XX:+UseG1GC \
  -XX:MaxGCPauseMillis=200 \
  -XX:+HeapDumpOnOutOfMemoryError \
  -XX:HeapDumpPath=/var/log/heapdump.hprof \
  -Xlog:gc*:file=/var/log/gc.log:time,uptime,level,tags:filecount=10,filesize=10M \
  -XX:MetaspaceSize=256m \
  -XX:MaxMetaspaceSize=512m \
  -Djava.security.egd=file:/dev/./urandom \
  -jar your-app.jar

Notes: -Xms=-Xmx avoids resize; G1GC default in Java 11+ for large heaps/low latency; heap dump on OOM; GC log rotation; metaspace caps; urandom avoids /dev/random blocking.

GC Log Analysis Commands

# Key metrics
grep -E "GC|Heap" /var/log/gc.log | awk '
  /Full GC/ {full_gc++ ; full_gc_time+=$NF}
  /GC/ {ygc++ ; ygc_time+=$NF}
  END {print "YGC:" ygc, "YGC time:" ygc_time "s", "FGC:" full_gc, "FGC time:" full_gc_time "s"}
'
# Pause distribution (ms)
grep "GC pause" /var/log/gc.log | awk '{print $NF}' | sort -n | awk '
  BEGIN{c=0} {a[c++]=$1}
  END {print "P50:", a[int(c*0.50)]; print "P90:", a[int(c*0.90)]; print "P99:", a[int(c*0.99)]; print "MAX:", a[c-1]}
'
# Longest pauses
grep -E "Full GC|GC pause" /var/log/gc.log | sort -kNF -t' ' | tail -5

Connection Pool (HikariCP / Spring Boot)

spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 3000
      idle-timeout: 600000
      max-lifetime: 1800000
      connection-test-query: SELECT 1

Too large → DB pressure, memory overhead. Too small → queuing, idle connections reaped by DB. Estimate from cores & target QPS, then load-test.

G1GC Tuning Reference

java -Xms8g -Xmx8g \
  -XX:+UseG1GC \
  -XX:MaxGCPauseMillis=200 \
  -XX:G1HeapRegionSize=8m \
  -XX:InitiatingHeapOccupancyPercent=45 \
  -XX:G1ReservePercent=10 \
  -XX:ConcGCThreads=4 \
  -XX:+ParallelRefProcEnabled \
  -XX:+UnlockExperimentalVMOptions \
  -XX:G1MixedGCLiveThresholdPercent=85 \
  -XX:G1HeapWastePercent=5 \
  -jar app.jar

Key: pause target 200ms; region 8M (2-32M power of 2); start concurrent mark at 45% occupancy; reserve 10% for to-space; concurrent threads ~cores/4; parallel ref processing; mixed GC thresholds.

Risk Warnings

Don't restart JVM lightly – loses thread stacks, GC state, heap layout; root cause may persist and recur instantly.

strace kills performance – 10x+ slowdown; use -c stats mode, not line-by-line -f.

jmap -dump triggers Full GC – do off-peak or use Arthas heapdump (lower overhead).

Heap dump I/O storm – 8G heap → 8G file; can stall process if disk saturated. Write to separate disk or tmpfs.

Rate-limit & circuit-break early – if root cause not quickly fixable, shed load at gateway (Nginx limit_req) to prevent cascade.

Preserve evidence – before restart/fix, save jstack, GC logs, Arthas output, top snapshots for postmortem.

Verbose GC logs can be huge – high QPS services may generate GBs/day; configure rotation and monitor disk.

Verification After Fix

CPU Returns to Normal

watch -n 2 "kubectl top pods -n order-service | grep order-api"
watch -n 2 "top -bn1 | head -20"
# Target: single instance 30-50%

GC Healthy

watch -n 5 "jstat -gcutil <PID> | tail -1"
# Old gen <50% and stable; Full GC frequency drops from 30s to minutes

Latency Restored

curl -o /dev/null -s -w "Response time: %{time_total}s
" http://localhost:8080/api/health
for i in {1..20}; do curl -o /dev/null -s -w "%{time_total}
" http://localhost:8080/api/health; done | sort | awk 'BEGIN{c=0} {a[c++]=$1} END{print "P99:", a[int(c*0.99)]}'
# P99 < 500ms typical

Thread Pool Drain

# Arthas
thread | grep "BLOCKED\|TIMED_WAITING" | wc -l
# jstack
jstack <PID> | grep -c "State:.*BLOCKED"
# Counts back to baseline

DB Queries Normal

mysql -e "SHOW FULL PROCESSLIST\G" | grep -v "Sleep" | head -20
mysql -e "SHOW SLAVE STATUS\G" | grep "Seconds_Behind_Master"
# Should be <30s

Health Checks Pass

kubectl get pods -n production | grep -E "Running|Healthy"
kubectl get pods -n production -o wide
# All pods Ready

Rollback Procedures

JVM Parameters

jinfo -flags <PID> > /tmp/jinfo-backup-$(date +%Y%m%d%H%M%S).log
# Revert startup script -Xmx etc.
kubectl rollout restart deployment/order-service -n production

Application Code

kubectl rollout history deployment/order-service -n production
kubectl rollout undo deployment/order-service -n production
kubectl rollout undo deployment/order-service -n production --to-revision=3
kubectl rollout status deployment/order-service -n production

Rate Limiting (Nginx)

limit_req_zone $binary_remote_addr zone=api_limit:10m rate=100r/s;
location /api/ {
  limit_req zone=api_limit burst=200 nodelay;
  proxy_pass http://backend;
}
nginx -s reload

Circuit Breaker (Sentinel/Hystrix)

# Sentinel Dashboard or API
curl http://localhost:8719/circuitbreakers/<ruleId>/status
# Hystrix via Actuator
curl http://localhost:8080/actuator/hystrix.stream | head -50

Additional CPU Fault Scenarios

1. Regex Catastrophic Backtracking

Pattern like (a+)+b on input aaaa...xa explodes. Find via flame graph or grep Pattern.compile / re.compile. Fix: possessive quantifiers, avoid nested quantifiers.

2. Excessive Serialization/Deserialization

Creating new ObjectMapper per request burns CPU. Reuse a static instance.

3. Infinite Loops / Recursion

Buggy equals with circular references → infinite recursion. Loop condition inverted ( continue instead of break).

4. Frequent GC

Check jstat -gcutil over 60s: YGC >10/min or any FGC → increase heap, switch GC (ZGC), reduce allocation rate (object pools, caching).

5. JNI Native CPU

jstack | grep -i "JNI\|native"

. Inspect native code for heavy loops, blocking calls, leaks causing GC pressure.

6. Lock Contention Spin

jstack | grep -B 5 "waiting for monitor"

. Many threads on same lock → reduce granularity, use concurrent collections, read-write locks.

7. Network I/O Softirq

cat /proc/softirqs

– high NET_RX/TX. Fix: more NIC queues, multi-queue NIC, enable RPS.

8. Crypto Overhead

HTTPS/AES/RSA heavy. Check perf stat -e 'crypto'. Use AES-NI hardware, TLS 1.3, session tickets.

9. Python GIL Contention

ps -eLf | grep <PID> | wc -l

. Many threads, low CPU → GIL thrashing. Use multiprocessing, C extensions, or PyPy.

10. Slow Queries Driving CPU

Usually I/O wait, but lock waits, full scans, temp tables can spike CPU. Check SHOW FULL PROCESSLIST, slow query log, information_schema.processlist.

Summary

CPU 100% troubleshooting is about peeling layers to find the root cause, not staring at the CPU number.

Closed-loop Investigation:

Confirm blast radius – single host or cluster? Business impact?

Identify process – top / pidstat; Java/Python/Go/system?

Classify CPU source – us (compute), sy (syscalls), wa (I/O). Java: check GC first.

Trace hotspots – jstack /Arthas for threads, perf for functions, strace for syscalls.

Pinpoint root cause – GC thrash, infinite loop, lock contention, downstream timeout, regex backtrack, serialization, etc. Each demands a different fix.

Fix & verify – CPU drops, GC normal, latency restored, thread pool healthy, DB queries fast, health checks green.

Postmortem – preserve artifacts, analyze root cause, create lasting actions, update alerts & runbooks.

Golden rule: CPU is the symptom, not the disease. Optimizing CPU metrics directly never works; find the code or thread actually burning cycles or waiting on I/O, and cure that.