When Linux CPU spikes to 99%, these 6 commands saved me three times

Establish the meaning of “99%”

CPU usage shown as 99% does not equal the top percentage; first check the load average relative to the number of cores using uptime, nproc or getconf _NPROCESSORS_ONLN. A load of 16 on a 4‑core machine indicates a clear queue, while the same number on a 64‑core box may be normal.

Break down CPU composition with top

Run top -b -n 1 -w 200 | head -n 40 and examine the us (user), sy (system), wa (iowait) and st (steal) columns. High us points to user‑space computation, high sy to system calls, high wa to I/O, and high st to virtualization steal.

Capture a CPU‑sorted process snapshot with ps

Use

ps -eo pid,ppid,user,stat,ni,%cpu,%mem,etime,comm,args --sort=-%cpu | head -n 30

. The D state means uninterruptible sleep; many D tasks mean load is not purely CPU‑bound.

Observe trends with pidstat

Run pidstat -u -r -d -h 1 10 (requires the sysstat package). The -u flag shows CPU, -r shows page faults, -d shows I/O. Continuous sampling avoids misreading transient spikes.

Detect single‑core saturation using mpstat

mpstat -P ALL 1 5

reveals which cores are fully utilized. When %soft or %irq are high, investigate network or interrupt sources before scaling the application.

Review historical peaks with sar

sar -u ALL -f /var/log/sa/sa$(date +%d)

shows when and how long a spike lasted, but it cannot directly point to the offending code.

Eliminate I/O, memory and virtualization mis‑interpretations

Read raw counters: grep -E '^(cpu|procs_running|procs_blocked)' /proc/stat and compute differences.

Check the run‑queue and blocked tasks with vmstat 1 10; high r means runnable tasks, high b means blocked tasks.

Inspect memory pressure: free -h and

grep -E 'MemAvailable|SwapTotal|SwapFree|Dirty|Writeback' /proc/meminfo

. Persistent swap usage can increase latency.

Observe context switches: vmstat -w 1 10 and grep -E '^(ctxt|intr)' /proc/stat. High cs suggests thread contention or lock storms.

Check soft‑interrupts and IRQs: cat /proc/softirqs and cat /proc/interrupts | head -n 30. Correlate %soft spikes with packet loss or latency.

Detect virtualization steal: mpstat 1 5 | awk '/all/ {print}' and systemd-detect-virt. Rising %steal means the vCPU is not scheduled.

Process, thread, and cgroup investigation

Identify the hot PID:

ps -p "$PID" -o pid,ppid,user,stat,ni,psr,%cpu,%mem,etime,cmd

and verify the executable path with readlink -f "/proc/$PID/exe". Confirm the thread causing the spike with top -H -b -n 1 -p "$PID" | head -n 30 and

ps -L -p "$PID" -o pid,tid,psr,stat,%cpu,comm --sort=-%cpu | head -n 20

. Record thread‑level changes via pidstat -u -w -t -p "$PID" 1 15.

Inspect process resource state:

grep -E 'State|Threads|VmRSS|voluntary_ctxt_switches|nonvoluntary_ctxt_switches' /proc/$PID/status

and count open file descriptors with ls /proc/$PID/fd | wc -l. High FD count may indicate leaks.

Check connections: sudo lsof -nP -p "$PID" | head -n 100 and sudo ss -ntp | rg "pid=$PID,". Correlate connection counts with request QPS and fd usage.

Verify cgroup limits: cat /proc/$PID/cgroup and cat /sys/fs/cgroup/cpu.max (cgroup v1/v2 differ). For throttling, examine throttled_usec growth in cpu.stat.

Container and Kubernetes context

For Docker containers, use docker ps --no-trunc, docker stats --no-stream <container_name>, and

docker inspect --format '{{.State.Pid}} {{.HostConfig.NanoCpus}} {{.HostConfig.CpuQuota}}' <container_name>

. Compare container CPU usage with host pidstat output.

In Kubernetes, ensure request and limit are set:

kubectl -n <namespace> top pod <pod_name> --containers

, kubectl -n <namespace> describe pod <pod_name>, and

kubectl -n <namespace> get pod <pod_name> -o jsonpath='{.spec.containers[*].resources}'

. kubectl top (metrics‑server) shows actual usage, but combine with cgroup stats for throttling evidence.

Runtime evidence and mitigation

Java: capture thread stacks with jcmd "$PID" Thread.print > /var/tmp/jstack-$PID.txt and search for nid=0x patterns.

Go: obtain a controlled pprof profile with

curl -fsS http://127.0.0.1:PPROF_PORT/debug/pprof/profile?seconds=30 -o /var/tmp/cpu.pprof

then go tool pprof -top /var/tmp/cpu.pprof. Ensure the port belongs to a trusted local service.

Native profiling: sudo perf top -p "$PID" (watch for missing symbols) and

sudo perf record -F 99 -p "$PID" -g -- sleep 30; sudo perf report --stdio | head -n 80

for reproducible reports.

System call tracing:

sudo timeout 10 strace -ff -tt -T -p "$PID" -o /var/tmp/strace-$PID

provides clues but should not be left running on high‑traffic services.

Align service logs with CPU windows: systemctl status SERVICE_NAME --no-pager and

journalctl -u SERVICE_NAME --since '15 minutes ago' --no-pager | tail -n 200

. Only when log events (OOM, errors, connection failures) coincide with resource spikes should they be considered root causes.

Check scheduled jobs and recent releases: systemctl list-timers --all, crontab -l, and grepping deployment logs for keywords like cron|systemd|deploy|release. Correlation does not imply causation; verify the actual PID and resource changes.

Graceful reloads should be used only when the service supports them: sudo systemctl reload SERVICE_NAME followed by health‑check curl -fsS http://127.0.0.1:PORT/HEALTH_PATH >/dev/null. If reload fails, roll back or isolate the instance.

Terminating a process is a last resort: capture full process info, send SIGTERM, wait, verify exit, and only use SIGKILL after confirming no data loss and after draining traffic.

Make the next troubleshooting faster

Persist a comprehensive evidence package (CPU composition, load, context switches, per‑core soft‑interrupts, process/container CPU, cgroup throttling, request latency, error rate) in a tarball generated by a script. Verify post‑remediation health for at least one normal business cycle before declaring the issue resolved.

#!/usr/bin/env bash
set -euo pipefail
OUT="/var/tmp/cpu-incident-$(date +%F-%H%M%S)"
mkdir -p "$OUT"
uptime > "$OUT/uptime.txt"
mpstat -P ALL 1 3 > "$OUT/mpstat.txt"
ps -eo pid,ppid,user,stat,%cpu,%mem,etime,args --sort=-%cpu > "$OUT/ps.txt"
vmstat 1 3 > "$OUT/vmstat.txt"
 tar -C "$(dirname "$OUT")" -czf "$OUT.tgz" "$(basename "$OUT")"

Collecting the package may contain internal addresses and command‑line parameters; restrict access. After closing the loop, verify business health, error rate, and resource curves, and document trigger conditions, rollback procedures, and monitoring gaps.

Use evidence to distinguish four high‑CPU scenarios

Same 99% can stem from completely different problems. The correct approach is to build a chain: symptom → command output → verification . The following scenarios guide the next step.

User‑space computation hotspot

If top and mpstat show sustained high us, pidstat -t points to the same thread, and JVM stacks or perf samples repeatedly land in the same function, the focus narrows to algorithmic inefficiency, serialization, compression, encryption, regex backtracking, or batch processing. Multiple workers must be examined; a single high‑CPU process is insufficient evidence.

PID=<PID>
for i in 1 2 3 4 5; do
  date -Is
  ps -p "$PID" -o pid,stat,%cpu,%mem,etime,cmd
  sleep 5
done

Continuous sampling confirms a persistent hotspot; short spikes may be JIT, GC, or normal batch tasks.

System‑state and network soft‑interrupt hotspot

High sy often accompanies frequent system calls, network packets, log writes, lock contention, or kernel work. Correlate with process, connection, soft‑interrupt, and network error metrics. sy alone does not prove “network overload”.

ss -tan | awk 'NR>1 {state[$1]++} END {for (s in state) print s, state[s]}' | sort
ss -s

Track SYN‑RECV, TIME‑WAIT, ESTAB increments against request QPS, fd usage, and load‑balancer logs. Connection counts alone are not incidents.

ip -s link show
cat /proc/net/dev

Interface errors, drops, and discards must be sampled twice to see increments; cumulative values since boot are insufficient.

nstat -a | rg 'Tcp(ActiveOpens|PassiveOpens|RetransSegs|InSegs|OutSegs)'
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog

iowait and uninterruptible tasks

High wa often means many tasks are waiting on disk, network file systems, or block storage. Adding vCPU rarely helps; first identify D state tasks and the affected mount points.

ps -eo pid,ppid,stat,wchan:32,comm,args | awk '$3 ~ /^D/ {print}' | head -n 50

wchan

shows the kernel wait location, not the final root cause. For NFS, cloud disks, or database data directories, also check server‑side and storage‑platform metrics.

findmnt -T <mount_path>
df -hT <mount_path>
journalctl -k --since '30 minutes ago' --no-pager | rg -i 'ext4|xfs|nfs|blk|i/o error' | tail -n 100

Container CPU quota throttling

In containers, “100% CPU” often means the process has exhausted its vCPU limit, even if the host has spare capacity. Options are: increase the limit, scale pod replicas, or optimise code after profiling.

kubectl -n <namespace> get pod <pod_name> -o wide
kubectl -n <namespace> get pod <pod_name> -o jsonpath='{range .spec.containers[*]}{.name}	{.resources.requests.cpu}	{.resources.limits.cpu}
'

To verify throttling, read the cgroup cpu.stat before and after a 30‑second interval and ensure throttled_usec grows continuously.

PID=<PID>
CG=$(awk -F: '$1=="0" {print $3}' /proc/$PID/cgroup)
cat "/sys/fs/cgroup${CG}/cpu.stat"
sleep 30
cat "/sys/fs/cgroup${CG}/cpu.stat"

Make the next troubleshooting faster

Host monitoring should retain CPU composition, load, context switches, per‑core soft‑interrupts, process/container CPU, cgroup throttling, request latency, and error rate. Example Prometheus queries (adjust metric names to your exporters):

100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))

This query reflects overall busy‑ness but cannot separate user, system, or iowait.

100 * avg by (instance) (rate(node_cpu_seconds_total{mode="iowait"}[5m]))

If iowait rises, trigger storage and downstream checks rather than an automatic restart.

sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total[5m]))
/ clamp_min(sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total[5m])), 1)

cAdvisor label names may differ; throttling ratios must be analysed together with CPU limits, latency, and actual load.

Post‑mortem conclusions should state the time window, CPU composition, hotspot thread or cgroup, related releases or dependency events, and verification steps. If evidence is insufficient, record the hypothesis as “to be validated” rather than a definitive fix.

After remediation (e.g., scaling, traffic migration, limit increase, or restart), continue observing for at least one full business cycle: request latency, error rate, queue depth, and I/O metrics must all improve. If CPU merely shifts between instances or errors drop while latency grows, the root cause remains unresolved.

For periodic spikes, ensure sampling covers pre‑spike, spike, and post‑spike periods. Low‑overhead host metrics and threshold‑triggered process snapshots help capture elusive incidents while respecting resource and privacy budgets.

In routine drills, prepare read‑only diagnostic accounts, controlled perf / jcmd permissions, designated collection directories, and automatic cleanup. All state‑changing actions (scaling, traffic migration, quota changes, reloads, kills) must document impact scope, pre‑checks, gradual rollout, success criteria, and rollback path; read‑only checks stay lightweight to avoid adding risk.

The goal is not to memorize more commands but to make evidence‑driven, reversible decisions under pressure.