How to Diagnose a 100% CPU Spike in 3 Minutes: From Symptom to Root Cause

0–3 Minutes: Confirm Impact Scope

Determine whether high CPU usage is confined to a single instance, node, pod, or the entire service. Distinguish CPU metrics (user, system, iowait, steal, softirq) from load average, which also includes waiting tasks and cannot be directly equated to CPU capacity.

date -u +'%FT%TZ'
uptime
nproc

# Show threads sorted by CPU usage, observe several rounds to avoid a single spike
 top -H -b -n 3 -d 1 | head -n 80

# Sample per‑CPU utilization and iowait
 mpstat -P ALL 1 10

# List top processes by CPU, RSS helps spot memory‑related issues
 ps -eo pid,ppid,user,stat,pcpu,pmem,rss,etime,cmd --sort=-pcpu | head -n 21

On multi‑core hosts a single process may exceed 100 % because it uses multiple logical CPUs. Record its PID, start parameters, start time, and parent process to avoid mis‑identifying cron jobs, backups, log compression, or security scans as business processes.

PID='PID'
ps -p "$PID" -o pid,ppid,user,lstart,etime,pcpu,pmem,args
tr '\0' ' ' < "/proc/$PID/cmdline"
echo
readlink -f "/proc/$PID/exe"

# Systemd service status, recent logs, and recent restarts
 sudo systemctl status 'SERVICE_NAME' --no-pager
 sudo journalctl -u 'SERVICE_NAME' --since '15 minutes ago' --no-pager | tail -n 300

# Observe run queue, context switches, I/O and page faults
 vmstat 1 10
 iostat -xz 1 10

If iowait is high, investigate disk latency, remote storage, write amplification, or backup tasks. A high steal indicates host‑level resource contention in virtualized environments. A high softirq suggests network packet storms. Do not simply add more CPU in these cases.

# Show connection counts and TCP states (read‑only)
 ss -s
 ss -tan | awk 'NR>1 {state[$1]++} END {for (s in state) print s, state[s]}' | sort

# Search recent logs for errors, timeouts, retries, OOM, rate‑limit, etc.
 journalctl -u 'SERVICE_NAME' --since '15 minutes ago' --no-pager | rg -i 'timeout|retry|exception|error|oom|rate limit' | tail -n 200

Process Located: Find Threads, Call Stacks, and System Calls

When a single process consumes CPU, narrow down to the hot thread. Keep sampling short and low‑impact; the debugging tool itself can add overhead. For latency‑sensitive services, run on a single gray‑scale instance and note start/end timestamps.

# Show top threads of the process
 PID='PID'
 top -H -b -n 1 -p "$PID" | head -n 40
 ps -L -p "$PID" -o pid,tid,pcpu,stat,comm --sort=-pcpu | head -n 20

# Verify perf availability and capture a short profile
 sudo perf --version
 sudo perf top -p 'PID'
 sudo perf record -F 99 -p 'PID' -g -- sleep 30
 sudo perf report --stdio | head -n 120

In the perf report, the hot functions are the evidence. If the hot spots are in JSON serialization, regex, compression, or encryption, the optimization path differs from a database bottleneck. If they are in network I/O or lock contention, further verification of the call path is required.

# Short‑term strace to capture network, file, and process syscalls (adds overhead)
 sudo timeout 15 strace -f -tt -T -p 'PID' -e trace=network,desc,process 2>&1 | tail -n 200

Strace can confirm loops involving connect, poll, read, write, futex or retry failures, but should not run continuously on a production process. Correlate its output with logs and metrics.

# List open file descriptors and check for leaks or stray subprocesses
 sudo lsof -p 'PID' | head -n 100
 ls -l "/proc/PID/fd" | wc -l

# Inspect cgroup constraints (path varies with cgroup v1/v2)
 cat /proc/PID/cgroup
 systemd-cgls --no-pager | head -n 120

Language‑Specific Evidence (Java, Python, Go)

For JVM services, high‑CPU threads appear as high‑CPU TIDs in thread dumps. Convert decimal TID to hex and match the nid in jstack. Preserve thread dumps according to data policies.

# Capture a Java thread dump
 PID='JAVA_PID'
 top -H -b -n 1 -p "$PID" | head -n 30
 jcmd "$PID" Thread.print > "threads-$(date -u +%Y%m%dT%H%M%SZ).txt"
 # Take multiple snapshots for reliability
 for i in 1 2 3; do
   jcmd "$PID" Thread.print > "threads-$i.txt"
   sleep 10
done

# Inspect GC, heap, and VM flags
 jcmd 'JAVA_PID' GC.heap_info
 jcmd 'JAVA_PID' VM.flags
 jstat -gcutil 'JAVA_PID' 1000 10

If high CPU coincides with frequent Full GC, verify allocation rate, heap size, object lifetimes, and leaks before enlarging thread pools. Heap dumps are risky and may contain sensitive data; obtain approval and ensure sufficient disk space.

# Python services: confirm interpreter, threads, and install py‑spy if needed
 ps -p 'PYTHON_PID' -o pid,args
 py-spy top --pid 'PYTHON_PID'

# Go services with protected pprof endpoint
 curl --fail --silent --show-error 'http://127.0.0.1:PPROF_PORT/debug/pprof/profile?seconds=30' -o cpu.pprof
 go tool pprof -top cpu.pprof

Never expose pprof endpoints publicly; only access them after confirming with the service owner and respecting network policies.

Control Impact: Rate‑Limit, Scale, Isolate, and Rollback

Remedial actions must be evidence‑driven. If traffic is high but request latency is normal, consider ingress rate‑limiting or horizontal scaling. For a single instance dead‑loop, detach it from the load balancer before restarting. If database timeouts trigger a retry storm, limit retries and recover dependencies instead of unbounded scaling.

# Gradually remove a pod from a Kubernetes Service
 kubectl -n 'NAMESPACE' get pod 'POD_NAME' --show-labels
 kubectl -n 'NAMESPACE' label pod 'POD_NAME' 'TRAFFIC_LABEL_KEY-' --overwrite

# Progressive scaling
 CURRENT=$(kubectl -n 'NAMESPACE' get deploy 'DEPLOYMENT_NAME' -o jsonpath='{.spec.replicas}
')
 kubectl -n 'NAMESPACE' scale deploy 'DEPLOYMENT_NAME' --replicas='NEW_REPLICA_COUNT'
 kubectl -n 'NAMESPACE' rollout status deploy/'DEPLOYMENT_NAME' --timeout=5m

# Nginx ingress rate‑limit example (zone must be defined in http block)
 limit_req_zone $binary_remote_addr zone=api_per_ip:10m rate=20r/s;
 location /api/ {
     limit_req zone=api_per_ip burst=40 nodelay;
     proxy_pass http://UPSTREAM_NAME;
 }

Before applying configuration changes, back up files, run syntax checks, and perform a gray‑scale reload. Define whitelists, error codes, retry behavior, and understand business impact.

# Controlled systemd restart after verification
 sudo systemctl restart 'SERVICE_NAME'
 sudo systemctl is-active 'SERVICE_NAME'
 sudo journalctl -u 'SERVICE_NAME' --since '5 minutes ago' --no-pager | tail -n 100

Restart impact depends on statefulness, queue ownership, connection migration, and idempotency. Verify healthy instance capacity and session strategy; if restart cannot maintain availability, use traffic draining or rolling updates.

Acceptance

CPU drop alone does not close the incident. Validate request success rate, P95/P99 latency, run‑queue length, I/O wait, error/retry counts, downstream latency, and whether temporary rate‑limit or scaling is still required.

# Simple verification loop
 for i in 1 2 3 4 5; do
   date -u +'%FT%TZ'
   ps -p 'PID' -o pid,pcpu,pmem,etime,args
   vmstat 1 1 | tail -n 1
   sleep 60
done

# Verify pod metrics and recent logs
 kubectl -n 'NAMESPACE' top pod -l 'app=APP_LABEL' --containers
 kubectl -n 'NAMESPACE' logs deploy/'DEPLOYMENT_NAME' --since=10m | rg -i 'panic|fatal|timeout|exception|error' | tail -n 100

After the event, gradually roll back temporary scaling, rate‑limit, or debug switches, observing at least one full business cycle after each step. Document the timeline, hot process/thread, call stack/metrics, root‑cause evidence, impact scope, actions taken, and rollback results. If no logs, metrics, config diff, or stack trace support the hypothesis, it must remain a hypothesis, not a definitive root cause.