Linux CPU at 99%? 6 Troubleshooting Commands That Saved Me 3 Times

Introduction

When CPU hits 99%, don't reboot immediately and don't kill processes based solely on top. The spike could be user-space computation, kernel system calls, I/O wait, soft interrupts, VM steal time, or container CPU quota throttling. This article builds a troubleshooting chain around six primary commands — uptime, top, ps, pidstat, mpstat, sar — then uses thread, cgroup, log, and runtime evidence to pinpoint the root cause. Placeholders like <PID>, <service>, <namespace>, <log-dir> are used throughout.

1. First, Interpret What "99%" Means

1.1 uptime : Load Must Be Viewed Relative to Core Count

Load average is the average number of runnable or uninterruptible tasks, not a CPU percentage. A load of 16 on a 4-core machine indicates queue buildup; the same number on a 64-core machine may be normal.

uptime
nproc
getconf _NPROCESSORS_ONLN

1.2 top : Break Down CPU Composition

Run top -b -n 1 -w 200 | head -n 40 to see the CPU breakdown: us high → computation-bound sy high → system-call heavy wa high → investigate I/O first st high → check virtualization contention

1.3 ps : Save a Process Snapshot Sorted by CPU

ps -eo pid,ppid,user,stat,ni,%cpu,%mem,etime,comm,args --sort=-%cpu | head -n 30

D

state means uninterruptible sleep; many D-state tasks mean high load may not be CPU-related.

1.4 pidstat : Observe Over 10 Seconds, Not a Single Instant

From sysstat package; avoid installing packages during an incident.

pidstat -u -r -d -h 1 10

-u

CPU, -r page faults, -d I/O. Sustained hotspots and short-lived tasks show different patterns.

1.5 mpstat : Detect Single-Core Saturation

mpstat -P ALL 1 5

If a few cores show high %soft or %irq, investigate network and interrupts first; don't just scale the application.

1.6 sar : Look Back at When the Spike Occurred

sar -u ALL -f /var/log/sa/sa$(date +%d)

Historical file path varies by distribution. It answers "when, how long, CPU composition" but cannot directly point to business code.

2. Rule Out I/O, Memory, and Virtualization Misdiagnoses

2.1 Read Raw Counters from /proc/stat

grep -E '^(cpu|procs_running|procs_blocked)' /proc/stat
head -n 12 /proc/stat

CPU usage requires comparing two samples; single jiffies are cumulative since boot.

2.2 Check Run Queue and Blocked Tasks with vmstat

vmstat 1 10

r

high = runnable tasks queued; b persistently non-zero = blocked. Correlate with us/sy/wa to attribute cause.

2.3 Exclude Memory Reclaim and Swap

free -h
grep -E 'MemAvailable|SwapTotal|SwapFree|Dirty|Writeback' /proc/meminfo

Focus on MemAvailable. Swap usage isn't necessarily a fault, but sustained swap-in/out raises latency and system overhead.

2.4 Observe Context Switches

vmstat -w 1 10
grep -E '^(ctxt|intr)' /proc/stat

High cs often stems from too many threads, lock contention, or short-connection storms; it's a clue, not a root cause.

2.5 Check Softirqs and IRQs

cat /proc/softirqs
cat /proc/interrupts | head -n 30

NET_RX

growth only supports softirq root cause when combined with single-core %soft, packet drops, and latency.

2.6 Investigate VM Steal Time

mpstat 1 5 | awk '/all/ {print}'
systemd-detect-virt || true

Sustained high %steal means vCPU isn't getting scheduled. Save monitoring evidence for cloud/virtualization platform to investigate the host.

2.7 Verify Block Device Latency

iostat -xz 1 5
lsblk -o NAME,TYPE,SIZE,MOUNTPOINTS

await

, queue length, and %util must be judged alongside application I/O patterns; don't rely on a single "100%" figure.

2.8 Check for OOM and Kernel Errors

journalctl -k --since '30 minutes ago' --no-pager | rg -i 'oom|killed process|error'
dmesg -T | tail -n 100

Kernel logs are critical evidence for OOM, driver errors, etc. Sanitize command lines when collecting.

3. Processes, Threads, and Cgroups

3.1 Confirm the Hot PID's Identity

PID=<PID>
ps -p "${PID}" -o pid,ppid,user,stat,ni,psr,%cpu,%mem,etime,cmd
readlink -f "/proc/${PID}/exe"

Re-verify PID before critical actions because processes can exit and be reused.

3.2 Find the Hot Thread

PID=<PID>
top -H -b -n 1 -p "${PID}" | head -n 30
ps -L -p "${PID}" -o pid,tid,psr,stat,%cpu,comm --sort=-%cpu | head -n 20

TID is key for subsequent JVM stack and perf analysis; psr only shows the most recent CPU.

3.3 Record Thread-Level Changes

PID=<PID>
pidstat -u -w -t -p "${PID}" 1 15

High CPU with low context switches suggests computation or tight loops; high switches point to locks, I/O, or thread model issues.

3.4 Check Process Resource Status

PID=<PID>
grep -E 'State|Threads|VmRSS|voluntary_ctxt_switches|nonvoluntary_ctxt_switches' /proc/${PID}/status
ls /proc/${PID}/fd | wc -l

FD count near limit can cause connection failures and retry storms; verify actual limits and leak sources.

3.5 Examine Connection Types

PID=<PID>
sudo lsof -nP -p "${PID}" | head -n 100
sudo ss -ntp | rg "pid=${PID},"

Connection list may contain internal addresses; restrict access. Many short connections must be evaluated with ingress request volume and pool config.

3.6 View the Process's Cgroup

PID=<PID>
cat /proc/${PID}/cgroup
cat /sys/fs/cgroup/cpu.max 2>/dev/null || true

Cgroup v1/v2 files differ; follow the actual path in /proc/<PID>/cgroup to read limits.

3.7 Verify Quota Throttling

PID=<PID>
CG=$(awk -F: '$1=="0" {print $3}' /proc/${PID}/cgroup)
cat "/sys/fs/cgroup${CG}/cpu.stat" 2>/dev/null || true

Only continuously growing throttled_usec supports a throttling conclusion. On v1 hosts, read the corresponding controller path.

3.8 In Containers, First Map to Host PID

docker ps --no-trunc
docker stats --no-stream <container>
docker inspect --format '{{.State.Pid}} {{.HostConfig.NanoCpus}} {{.HostConfig.CpuQuota}}' <container>

Container snapshots must be cross-checked with host pidstat. Runtime commands need authorization; don't equate container usage with absence of throttling.

3.9 In Kubernetes, Check Requests and Limits

kubectl -n <namespace> top pod <pod> --containers
kubectl -n <namespace> describe pod <pod>
kubectl -n <namespace> get pod <pod> -o jsonpath='{.spec.containers[*].resources}'

kubectl top

relies on metrics-server; it shows usage, not throttling. Combine with cgroup stats and pod events.

4. Runtime Evidence and Mitigation

4.1 Java: Capture Thread Stacks

PID=<Java PID>
jcmd "${PID}" Thread.print > /var/tmp/jstack-${PID}.txt
grep -n 'nid=0x' /var/tmp/jstack-${PID}.txt | head -n 30

Convert hot TID to hex, then match the same nid. jcmd may affect the JVM; follow the application runbook.

4.2 Go: Use Controlled pprof Sampling

curl -fsS http://127.0.0.1:<pprof-port>/debug/pprof/profile?seconds=30 -o /var/tmp/cpu.pprof
go tool pprof -top /var/tmp/cpu.pprof

Ensure the port is a local, controlled service. 30-second sampling has overhead; run per capacity and security requirements in production.

4.3 Native Processes: Short perf top

PID=<PID>
sudo perf top -p "${PID}"

perf

is subject to kernel permissions, symbols, and security restrictions. Don't lower system security settings for temporary diagnosis.

4.4 perf record for Reproducible Reports

PID=<PID>
sudo perf record -F 99 -p "${PID}" -g -- sleep 30
sudo perf report --stdio | head -n 80

99 Hz for 30 seconds is a conservative example; missing symbols mean unknown cannot be interpreted as a business function.

4.5 strace Only for System-Call Clues

PID=<PID>
sudo timeout 10 strace -ff -tt -T -p "${PID}" -o /var/tmp/strace-${PID}

strace

perturbs high-frequency syscall processes; don't leave it attached to high-traffic services. Unsuitable for pure compute hotspots.

4.6 Align Service Logs with CPU Time Window

systemctl status <service> --no-pager
journalctl -u <service> --since '15 minutes ago' --no-pager | tail -n 200

Releases, OOM, connection failures, and retries are candidate root causes; they become conclusions only when correlated with resource curves, threads, or stacks.

4.7 Check Cron Timers and Recent Releases

systemctl list-timers --all
crontab -l 2>/dev/null || true
rg -n 'cron|systemd|deploy|release' <log-dir> 2>/dev/null | tail -n 100

Hourly correlation ≠ causation; still confirm actual PID, executed command, and resource change.

4.8 Graceful Reload Only When Draining and Replicas Exist

set -euo pipefail
sudo systemctl reload <service>
systemctl is-active --quiet <service>
curl -fsS http://127.0.0.1:<port>/<health-path> >/dev/null

reload

support is service-dependent; not a substitute for restart. On health-check failure, follow rollback procedure or drain the instance.

4.9 Killing the Process Is a Last Resort

PID=<PID>
ps -p "${PID}" -o pid,ppid,lstart,cmd
kill -TERM "${PID}"
sleep 10
ps -p "${PID}" -o pid,stat,cmd || true

SIGTERM

allows resource cleanup. SIGKILL only when process won't exit, data risk assessed, and approval granted; drain first, then verify error rates, queues, CPU, and data consistency.

4.10 Save an Evidence Package Before Any Restart, Release, or Mitigation

#!/usr/bin/env bash
set -euo pipefail

OUT="/var/tmp/cpu-incident-$(date +%F-%H%M%S)"
mkdir -p "${OUT}"
uptime > "${OUT}/uptime.txt"
mpstat -P ALL 1 3 > "${OUT}/mpstat.txt"
ps -eo pid,ppid,user,stat,%cpu,%mem,etime,args --sort=-%cpu > "${OUT}/ps.txt"
vmstat 1 3 > "${OUT}/vmstat.txt"
tar -C "$(dirname "${OUT}")" -czf "${OUT}.tgz" "$(basename "${OUT}")"

Collection may contain internal addresses and command-line args; restrict access. At closure, verify business health, error rates, and resource curves; record trigger conditions, rollback method, and monitoring gaps in the postmortem.

5. Use Evidence to Distinguish Four High-CPU Scenarios

The same 99% can be completely different problems. The correct approach is not to apply experience-based conclusions but to chain "symptom → command output → follow-up verification" into an evidence trail. The scenarios below guide next steps; they do not replace on-site data.

5.1 User-Space Compute Hotspot

If top and mpstat show sustained high us, pidstat -t points to the same thread, and JVM stacks or perf samples repeatedly land in the same function, then converge on algorithm, serialization, compression, encryption, regex backtracking, or batch processing. A single high process isn't enough: multi-worker services may normally distribute computation across child processes.

PID=<PID>
for i in 1 2 3 4 5; do
  date -Is
  ps -p "${PID}" -o pid,stat,%cpu,%mem,etime,cmd
  sleep 5
done

Continuous sampling script; add log rotation for production use. Five consecutive highs support sustained hotspot; brief spikes may be JIT, GC, or normal batch jobs.

PID=<PID>
taskset -cp "${PID}"
grep -E 'Cpus_allowed_list|Mems_allowed_list' /proc/${PID}/status

If the process is pinned to few CPUs, overall utilization may be low but queuing occurs. Changing CPU affinity affects NUMA, cache, and real-time tasks; during troubleshooting only collect evidence, don't unbind directly.

5.2 System/Kernel & Network Softirq Hotspot

High sy often comes from frequent syscalls, network packets, log writes, lock contention, or kernel work. Conclusion requires correlating process, connections, softirqs, and network errors. High sy alone doesn't mean "too much network".

ss -tan | awk 'NR>1 {state[$1]++} END {for (s in state) print s, state[s]}' | sort
ss -s

SYN-RECV, TIME-WAIT, or ESTAB increments must align with request QPS, FD usage, and load-balancer logs. High connection count itself isn't an incident.

ip -s link show
cat /proc/net/dev

Interface errors, drops, and discards need two samples to see deltas; cumulative since-boot values can't prove this spike's cause.

nstat -a | rg 'Tcp(ActiveOpens|PassiveOpens|RetransSegs|InSegs|OutSegs)'
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog

nstat -a

is read-only. Don't use -z to zero counters. Don't increase backlog just because of retransmits; first check application accept capacity, downstream timeouts, and ingress traffic.

5.3 IOWait & Uninterruptible Tasks

High iowait means the machine may not be compute-saturated, but many tasks wait for disk, network filesystem, or block storage. Adding vCPUs often doesn't help and may increase queuing. First confirm D-state tasks and mount points.

ps -eo pid,ppid,stat,wchan:32,comm,args | awk '$3 ~ /^D/ {print}' | head -n 50

wchan

is a kernel wait-location hint, not the final root cause. For NFS, cloud disks, DB data directories, also check server-side and storage platform metrics.

findmnt -T <mount-path>
df -hT <mount-path>
journalctl -k --since '30 minutes ago' --no-pager | rg -i 'ext4|xfs|nfs|blk|i/o error' | tail -n 100

Replace <mount-path> with the hot service's data directory. Full disk, read-only remount, and NFS stale handles can manifest as timeouts and high load.

5.4 Container Quota Throttling

In containers, "100% CPU" often means the container exhausted its 1 vCPU limit while the host still has spare capacity. Raising CPU limit, adding pod replicas, and optimizing code carry different risks: higher limit contends for node resources; scaling replicas depends on session and queue horizontal scalability; code optimization requires profiling proof.

kubectl -n <namespace> get pod <pod> -o wide
kubectl -n <namespace> get pod <pod> -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.requests.cpu}{"\t"}{.resources.limits.cpu}{"
"}{end}'

This only reads resource config. To change limits, first check namespace quota, node capacity, HPA behavior, and canary deployment plan; don't just fill node total CPU.

PID=<PID>
CG=$(awk -F: '$1=="0" {print $3}' /proc/${PID}/cgroup)
cat "/sys/fs/cgroup${CG}/cpu.stat"
sleep 30
cat "/sys/fs/cgroup${CG}/cpu.stat"

Throttling-related fields increasing between two samples supports quota throttling conclusion. Record sampling interval and corresponding business traffic to avoid mistaking historical counters from low-traffic periods for real-time issues.

6. Make the Next Troubleshooting Faster

Host monitoring should retain CPU breakdown, load, context switches, per-core softirqs, process/container CPU, cgroup throttling, request latency, and error rates. Prometheus metric names depend on actual exporter exposure; the queries below apply only when corresponding metrics exist.

100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))

This reflects overall busyness but cannot distinguish user, system, and iowait.

100 * avg by (instance) (rate(node_cpu_seconds_total{mode="iowait"}[5m]))

If iowait rises, trigger storage and downstream dependency investigation, not automatic application restart.

sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total[5m]))
/ clamp_min(sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total[5m])), 1)

cAdvisor metric labels may differ. Throttling ratio must be analyzed with CPU limit, latency, and actual load.

A troubleshooting conclusion should at least state impact window, CPU composition, hot thread or cgroup, related release/dependency events, and verification actions. When evidence is insufficient, record it as a "hypothesis pending verification" rather than writing a temporary mitigation as a root-cause fix.

The most commonly missed step in practice is "post-recovery validation." If service recovers via scaling, traffic migration, raising CPU limit, or restart, continue observing at least one full business cycle: has request latency returned? Is error rate stable? Are queues drained? Have accompanying metrics like retransmits and I/O improved? If CPU merely shifted to another instance, or error rate dropped but latency grew significantly, the problem isn't truly solved. Link troubleshooting commands, sampling times, version changes, capacity decisions, and final fix commits in a single postmortem so the next incident can quickly distinguish known patterns from new issues.

For periodic spikes, ensure sampling windows cover pre-spike, during-spike, and post-spike. Capturing only the recovery phase often yields a false "no anomaly" conclusion. For non-reproducible intermittent issues, retain low-overhead host metrics and threshold-triggered process snapshots, but limit collection frequency, retention period, and sensitive fields to avoid the diagnostic system itself exhausting CPU, disk, or privacy budget. When error rates and latency already visibly affect users, prioritize existing draining, scaling, or degradation switches; diagnostic actions must strike an auditable balance between business recovery and evidence completeness.

In routine drills, pre-provision read-only diagnostic accounts, controlled perf / jcmd permissions, explicit collection directories, and auto-cleanup mechanisms. Then engineers won't need to elevate permissions or copy unverified commands during a real spike. All actions that change service state — scaling, traffic migration, quota changes, service reloads, process kills — must document impact scope, pre-checks, canary method, success criteria, and rollback path; read-only checks stay lightweight to avoid turning the troubleshooting process into an additional risk source.

The ultimate goal isn't to memorize more commands, but to make reversible, evidence-based judgments under pressure.