Linux CPU Spike Troubleshooting: From Alert to Root Cause

Confirm Alert Scope

Monitoring CPU metrics can represent host utilization, per-core percentage, process cumulative time, container quota ratio, or load average. On an 8-core host a single saturated thread shows ~12.5% host utilization but 100% in top 's process view. Record UTC/local time, alert window, core count, and monitoring formula.

date --iso-8601=seconds
timedatectl status
nproc
lscpu
uptime

load average

reflects runnable + uninterruptible task queue length, not CPU percentage. High load may stem from disk I/O blocking; correlate with run queue and CPU states.

Observe overall CPU breakdown: mpstat -P ALL 1 10 Focus on %usr, %sys, %iowait, %irq, %soft, %steal, %idle. High %iowait means CPU idle with pending I/O, not compute exhaustion; high %steal on VMs indicates host contention.

Use vmstat for run queue, paging, context switches:

vmstat 1 10

r

consistently above logical CPUs with low idle supports CPU queuing; high b and wa point to I/O; non-zero si/so indicates paging pressure. Single samples are insufficient.

Find the Consuming Process

Sample per-process user/kernel/I/O/context-switch: pidstat -u -r -d -w 1 10 High %usr suggests computation, serialization, regex, or GC; high %system warrants syscall, network, kernel path inspection; many voluntary/involuntary context switches may indicate locks, excess threads, or scheduler contention.

Stable process snapshot sorted by CPU:

ps -eo pid,ppid,user,stat,psr,pcpu,pmem,etimes,comm,args \
  --sort=-pcpu | head -n 30

ps

%CPU

is lifetime average, possibly masking short spikes; complement with pidstat interval sampling. Sanitize command lines before saving.

Identify service ownership and cgroup:

systemctl status <service> --no-pager
systemctl show <service> \
  -p MainPID -p ControlGroup -p CPUUsageNSec -p CPUQuotaPerSecUSec
cat /proc/<pid>/cgroup

If MainPID differs from high-CPU PID, it may be a worker child, container, or external process. Do not kill based on name alone.

Drill Down to Threads

Multi-threaded processes require thread-level identification: pidstat -t -u -p <pid> 1 10 Only persistently high-CPU TIDs merit further sampling; transient threads may have exited. Record timestamps.

View thread names and scheduler state:

ps -L -p <pid> \
  -o pid,tid,psr,stat,pcpu,comm,wchan:32 --sort=-pcpu | head -n 30
cat /proc/<pid>/task/<tid>/status

wchan

is meaningful only for sleeping threads; running hot threads show -. Application-set thread names can directly map to thread pools, GC, or business workers.

Java thread dumps use hex nid. Convert TID:

printf '0x%x
' <tid>
jcmd <pid> Thread.print -l > <dir>/threads-$(date +%s).txt

Use JDK matching target JVM, run as same user. Thread dumps briefly enter safepoint and contain business stacks; control frequency and file permissions.

Capture three consecutive thread stacks to confirm stable hotspot:

#!/usr/bin/env bash
set -euo pipefail
PID="<pid>"
OUT_DIR="<dir>"
install -d -m 0700 "${OUT_DIR}"
for i in 1 2 3; do
  jcmd "${PID}" Thread.print -l > "${OUT_DIR}/thread-${i}.txt"
  if (( i < 3 )); then sleep 5; fi
done

Same thread RUNNABLE in same business function across multiple dumps constitutes code hotspot evidence; a single dump cannot rule out sampling chance.

Distinguish User, Kernel, and Interrupt

When system time is high, examine syscall summary. strace adds overhead; limit to 20 seconds, test on low-risk instance first: sudo timeout 20 strace -f -c -p <pid> Many futex calls may indicate lock waits or thread sync, not automatically root cause; many read/write/sendto/recvfrom need file descriptor, network, and stack correlation.

Check interrupt distribution:

watch -n 1 'cat /proc/interrupts | head -n 30'
cat /proc/softirqs

If a NIC queue interrupt grows rapidly on one core while %soft is high, inspect RSS/RPS, driver, traffic, and drops. Exit watch with Ctrl-C.

View NIC stats and softirq pressure:

ip -s link show dev <nic>
ethtool -S <nic> | grep -Ei 'drop|miss|error|timeout|queue'
sar -n DEV,EDEV 1 10

Driver stat fields vary; use actual output. Watch error counters as deltas, not absolute values.

Check irqbalance and interrupt affinity:

systemctl status irqbalance --no-pager
grep -H . /proc/irq/<irq>/smp_affinity_list
ethtool -l <nic>

<irq>

from /proc/interrupts. Manual affinity changes may conflict with irqbalance; treat as performance policy change, backup, canary, and prepare rollback.

Check Memory, I/O, and Reclaim Illusions

Memory pressure forces CPU into reclaim, compaction, or swap:

free -h
sar -B 1 10
sar -W 1 10
cat /proc/pressure/memory

Rising pgscan, pgsteal, swap in/out, and memory PSI together support reclaim pressure. High cache alone is not memory shortage; Linux actively uses page cache.

Check I/O latency and queues:

iostat -xz 1 10
cat /proc/pressure/io

High %util on parallel devices doesn't necessarily mean saturation; combine with await, queue depth, throughput, device type. If CPU alert coincides with high iowait, optimizing CPU code may be ineffective.

Examine kernel logs for OOM, hung task, hardware errors:

sudo journalctl -k --since '-30 minutes' --no-pager \
  | grep -Ei 'oom|out of memory|hung task|soft lockup|hard lockup|mce|thermal|throttl|rcu'

Log timestamps must align with alert window. Thermal throttling can make same workload consume more CPU time; check hardware sensors and platform monitoring.

Use perf to Find Function Hotspots

perf

requires matching kernel tools, symbols, and permissions. Verify version and kernel limits:

perf --version
uname -r
sysctl kernel.perf_event_paranoid kernel.kptr_restrict

Do not permanently lower security limits for sampling. If policy forbids perf, use language runtime profilers or reproduce in pre-production.

30-second, 99Hz call-stack sampling:

sudo perf record -F 99 -g -p <pid> -- sleep 30
sudo perf report --stdio --sort comm,dso,symbol | head -n 80

99Hz reduces sync bias with common timer frequencies. perf.data can be large with symbols; collect in controlled directory. Hot function share must align with process/thread CPU contemporaneous evidence.

Count only syscalls and scheduler events:

sudo perf stat -p <pid> \
  -e task-clock,context-switches,cpu-migrations,page-faults,cycles,instructions \
  -- sleep 20

Low IPC (instructions/cycles) may indicate cache misses, branch mispredicts, or stalls, but cannot alone determine root cause. Virtualized counters may be constrained.

If scheduler latency suspected, briefly record sched events:

sudo perf sched record -- sleep 10
sudo perf sched latency --sort max | head -n 40

This is system-wide; busy hosts incur higher overhead and data volume. Canary first; do not run continuously during sustained peaks.

Language Runtime Specific Evidence

Java: examine GC and compilation activity:

jstat -gcutil <pid> 1000 10
jcmd <pid> VM.flags
jcmd <pid> Compiler.queue
jcmd <pid> GC.heap_info

jstat

fields vary by GC algorithm. Frequent Full GC, high GC thread CPU, and heap occupancy changes must co-occur to support GC root cause; do not increase heap just because GC appears in logs.

Short Java Flight Recorder for production diagnosis, but has overhead; verify JDK version and licensing:

jcmd <pid> JFR.start \
  name=cpu-investigation settings=profile duration=60s \
  filename=<dir>/cpu-investigation.jfr
jcmd <pid> JFR.check

settings=profile

has higher overhead than default. JFR files may contain class names, paths, business events; secure transfer and storage.

Python: py-spy if installed and ptrace permitted per org policy. Limit sampling:

sudo timeout 30 py-spy top --pid <pid> --rate 50

Do not install unknown binaries in production. If tool unavailable, prefer built-in profiling or reproduce in isolation.

Container and cgroup Throttling

High CPU inside container requires quota verification. cgroup v2 cpu.stat shows usage and throttling:

CGROUP_PATH="$(systemctl show -p ControlGroup --value <service>)"
sudo cat "/sys/fs/cgroup${CGROUP_PATH}/cpu.max"
sudo cat "/sys/fs/cgroup${CGROUP_PATH}/cpu.stat"
sudo cat "/sys/fs/cgroup${CGROUP_PATH}/cpu.pressure"

cpu.max

max

means no quota; otherwise "quota period" in microseconds. Watch nr_throttled and throttled_usec deltas; high throttling doesn't imply high host CPU.

Kubernetes resource config and current usage:

kubectl -n <ns> get pod <pod> -o jsonpath='{.spec.containers[*].resources}{"
"}'
kubectl -n <ns> top pod <pod> --containers
kubectl -n <ns> describe pod <pod>

kubectl top

relies on Metrics Server, provides recent aggregates, unsuitable for reconstructing ended second-level spikes. Need Prometheus historical metrics.

Container CPU usage query (adjust to actual exporter/cAdvisor metrics):

sum by (namespace, pod, container) (
  rate(container_cpu_usage_seconds_total{
    namespace="<ns>",pod="<pod>",container!=""}
  [5m])
)

Result in CPU cores. For relative limit, join with kube_pod_container_resource_limits{resource="cpu"} and handle missing limits.

Throttling ratio (check actual exposed metrics):

sum(rate(container_cpu_cfs_throttled_periods_total{namespace="<ns>",pod="<pod>"}[5m]))
/
sum(rate(container_cpu_cfs_periods_total{namespace="<ns>",pod="<pod>"}[5m]))

Metric may vary across cgroup v2/cAdvisor versions; verify first. Before raising limit, confirm node allocatable CPU and neighbor impact.

Correlate Deployments, Traffic, and Cron Jobs

CPU rises often coincide with releases, config changes, cache invalidation, batch jobs, or traffic shifts. Check systemd logs and service start times:

systemctl show <service> -p ActiveEnterTimestamp -p ExecMainStartTimestamp -p MainPID
sudo journalctl -u <service> --since '-1 hour' --no-pager
sudo journalctl --since '-1 hour' _PID=<pid> --no-pager | tail -n 200

Log errors must be correlated with frequency and time; an error can be both cause and symptom of overload.

Check timers:

systemctl list-timers --all
sudo grep -RIn --exclude='*.bak' . /etc/cron.d /etc/cron.hourly 2>/dev/null
sudo crontab -l

Backups, compression, log rotation, AV scans, stats jobs may cause periodic CPU. Do not disable without confirming business purpose and catch-up plan.

Check recent installs and config changes:

sudo dnf history list | head -n 20
sudo find /etc -xdev -type f -mmin -120 -printf '%TY-%Tm-%Td %TH:%TM %p
' \
  2>/dev/null | sort

File timestamps only prove modification occurred, not causation. Root cause requires diff content, temporal correlation, and reproducible behavior.

Automated Scene Collection

Script collects 30-second baseline evidence without modifying system state; command lines and logs may contain sensitive data, output directory permissions 700:

#!/usr/bin/env bash
set -euo pipefail
OUT_DIR="<dir>/cpu-$(date -u +%Y%m%dT%H%M%SZ)"
DURATION=30
install -d -m 0700 "${OUT_DIR}"
date --iso-8601=seconds > "${OUT_DIR}/time.txt"
uptime > "${OUT_DIR}/uptime.txt"
lscpu > "${OUT_DIR}/lscpu.txt"
ps -eo pid,ppid,user,stat,psr,pcpu,pmem,etimes,comm,args --sort=-pcpu \
  > "${OUT_DIR}/processes.txt"
mpstat -P ALL 1 "${DURATION}" > "${OUT_DIR}/mpstat.txt" &
p1=$!
pidstat -u -r -d -w 1 "${DURATION}" > "${OUT_DIR}/pidstat.txt" &
p2=$!
vmstat 1 "${DURATION}" > "${OUT_DIR}/vmstat.txt" &
p3=$!
wait "${p1}" "${p2}" "${p3}"
journalctl -k --since '-15 minutes' --no-pager > "${OUT_DIR}/kernel.txt"
printf 'saved=%s
' "${OUT_DIR}"

Parallel sampling adds slight load; if host extremely busy, shorten duration or collect single items first. Script excludes perf, strace, packet capture due to separate risk assessment.

Compute checksums for integrity:

find <dir> -maxdepth 1 -type f -print0 \
  | sort -z \
  | xargs -0 sha256sum > <dir>/SHA256SUMS

Replace <dir> with exact generated directory to avoid scanning unrelated data.

Remediation Choices and Risks

If single-instance code hotspot confirmed, drain from load balancer, preserve scene, analyze on other instances if capacity allows. If all instances rise with traffic, prioritize rate limiting, caching, scaling; if downstream slowness causes retry storms, limit retries and restore dependency. Raising CPU quota only relieves throttling, cannot fix infinite loops.

Killing process is high-risk: interrupts requests, loses in-memory state, may be immediately restarted by systemd. Verify replicas, health checks, graceful stop timeout, persistent state. Prefer service manager:

sudo systemctl show <service> \
  -p MainPID -p Restart -p TimeoutStopUSec -p KillMode
sudo systemctl stop <service>
sudo systemctl status <service> --no-pager

Before execution, complete traffic drain and confirm capacity. After, verify requests, logs, ports, dependencies. Rollback is systemctl start <service> and re-add to traffic; if start fails, check journal, do not loop restart.

Temporary systemd CPU quota protects host but adds latency to service; must canary:

sudo systemctl set-property --runtime <service> CPUQuota=200%
systemctl show <service> -p CPUQuotaPerSecUSec
cat /proc/pressure/cpu

200%

≈ two CPU cores quota. --runtime reverts on reboot, suitable for emergency validation. Rollback with

systemctl set-property --runtime <service> CPUQuota=infinity

, then recheck service latency and host CPU.

If rolling back app version, save current version, config diff, thread stacks, profiler results. Kubernetes rollback example:

kubectl -n <ns> rollout history deployment/<deploy>
kubectl -n <ns> rollout undo deployment/<deploy> --to-revision=<rev>
kubectl -n <ns> rollout status deployment/<deploy> --timeout=10m
kubectl -n <ns> get pods -l app=<label> -o wide

Rollback recreates pods; check PDB, capacity, DB migration compatibility, irreversible data changes. If version includes non-backward-compatible schema migration, do not rollback binary alone.

Validate Root Cause, Not Just "CPU Dropped"

Sound conclusion must answer: which host/container, which process/thread, user vs kernel, which function/wait path, start time, correlated change, which metrics moved as expected after action. Example: "Java worker high CPU" is not root cause; "three thread dumps and 30s perf both point to same parser function, call volume rose after release, rollback restored function sample share and CPU simultaneously" is complete evidence chain.

After fix, repeat collection of mpstat, pidstat, thread/function evidence, business latency covering original peak window. Confirm pressure not shifted to I/O, DB, downstream, other nodes. Retain commands, time windows, versions, config diffs, rollback results to turn incident into alert thresholds, capacity planning, release validation rules.

When Process Gone or Spike Ended

After short spike, top and /proc/<pid> cannot reconstruct scene. Rely on historical monitoring, journald, process accounting, pre-deployed low-overhead continuous sampling. Do not dismiss alert based on current normal state; do not treat single monitoring point as full duration.

If sysstat enabled, query historical sar files:

sar -u -f /var/log/sa/sa<day> -s <start> -e <end>
sar -q -f /var/log/sa/sa<day> -s <start> -e <end>
sar -w -f /var/log/sa/sa<day> -s <start> -e <end>

<day>

usually two-digit day; path/format depend on sysstat config/version; use ls /var/log/sa and sar --help to confirm. Historical sampling granularity (e.g., 10 min) cannot prove 20-second spike internals.

Check if process killed by OOM, systemd, or human:

sudo journalctl --since '<start>' --until '<end>' --no-pager \
  | grep -Ei 'killed process|oom-kill|Main process exited|Stopped|Started|sudo|session opened'
last -x | head -n 30

last -x

from wtmp, may be rotated. Human attribution needs sudo/audit logs and change records, not just login time guesses.

If process accounting enabled, acct / psacct records exited process command and cumulative CPU, but enabling adds disk writes and only provides data after activation:

systemctl status psacct --no-pager
lastcomm | head -n 50
sa --sort-cpu | head -n 30

Tool installation, service name, log retention vary by distro. Cannot reconstruct thread stacks or function hotspots; only helps discover short-lived high-CPU commands.

Scheduler, NUMA, and CPU Frequency

On multi-socket NUMA, cross-node memory access increases latency and CPU cycles. Inspect topology and allocation before pinning:

numactl --hardware
numastat -p <pid>
taskset -cp <pid>

High remote memory ratio is a clue; still need application allocation and CPU affinity correlation. Manual taskset may break service manager, container orchestration, or irq distribution; apply via formal config, canary, record original values.

Check CPU frequency and governor:

cpupower frequency-info
grep -H . /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor 2>/dev/null | head

Some VMs don't expose cpufreq. Frequency drops may come from power policy, thermal, or platform limits; changing governor alters power/thermal design, requires platform team evaluation.

Scheduler latency may also come from real-time priority tasks. List policies and priorities:

ps -eLo pid,tid,cls,rtprio,pri,ni,psr,stat,pcpu,comm \
  --sort=-rtprio,-pcpu | head -n 40

FF

/ RR real-time threads misconfigured can starve normal tasks. Lowering real-time priority is behavioral change; confirm process purpose, watchdogs, vendor recommendations first.

Steal Time in VMs

VM %steal persistently rising means vCPU wanted to run but hypervisor descheduled. Adding replicas inside guest may worsen contention. Evidence must include mpstat / sar steal time, co-located VM trends, cloud platform host metrics, instance spec limits.

sar -u ALL 1 10
grep -E '^cpu ' /proc/stat
systemd-detect-virt

For burstable instances, check CPU credit/credit balance; dynamic metric from cloud monitoring. Without platform evidence, list steal as suspicion, cannot assert host oversubscription.

Packet Rate, Not Bandwidth

Small-packet high PPS can drive softirq CPU high while bandwidth low. Compare packet rate, byte rate, drops, softnet stats:

sar -n DEV 1 10
awk '{for (i=1;i<=NF;i++) printf "%s%s", strtonum("0x" $i), (i==NF?ORS:OFS)}' \
  /proc/net/softnet_stat | head

/proc/net/softnet_stat

field meanings per kernel docs; second column often used for drop cumulative; must watch delta and verify against current kernel. Above awk uses gawk strtonum; if unavailable, keep raw hex.

Limit 20-second packet header capture for protocol distribution:

sudo timeout 20 tcpdump -ni <nic> -s 128 -c 20000 \
  -w <dir>/cpu-spike.pcap

Packet capture involves sensitive data and consumes CPU/disk. -s 128 limits per-packet length, -c limits count; verify compliance, space, host headroom before execution. Delete or archive per retention policy after analysis.

Security Events and Anomalous Processes

Unknown high-CPU process could be runaway script or security incident. Do not immediately delete binary or wipe logs. First isolate risk, preserve hash, path, parent process, network connections, open files, then hand to security process.

sudo readlink -f /proc/<pid>/exe
sudo sha256sum /proc/<pid>/exe
sudo tr '\0' ' ' < /proc/<pid>/cmdline; echo
sudo lsof -nP -p <pid> | head -n 100
sudo ss -tpn | grep "pid=<pid>,"

Command line and connections may be sensitive. If process image deleted, /proc/.../exe shows (deleted); still do not terminate without forensics. If lateral movement or crypto-mining suspected, isolate network per incident response and preserve evidence.

Pre-position Observability for Next Spike

Second-level spikes need second-or-finer metrics. Prometheus host CPU query per actual node_exporter metrics:

100 * (1 - avg by (instance) (
  rate(node_cpu_seconds_total{mode="idle"}[2m])
))

Short window reacts fast but noisy; alert can require high utilization AND high run queue OR business latency degradation. Multi-core process hotspots need process exporter, eBPF profiler, or runtime metrics.

Record CPU pressure (metric names per actual exporter):

rate(node_pressure_cpu_waiting_seconds_total[5m])

PSI reflects time tasks wait for CPU, closer to "is CPU really scarce" than raw utilization. Different node_exporter versions may not expose it; check /metrics.

Production can run low-overhead collection script via systemd timer, but not heavy commands every second. Service and timer example:

# /etc/systemd/system/cpu-snapshot.service
[Unit]
Description=Collect a lightweight CPU snapshot

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/cpu-snapshot
Nice=10
IOSchedulingClass=idle

# /etc/systemd/system/cpu-snapshot.timer
[Unit]
Description=Run CPU snapshot every minute

[Timer]
OnBootSec=2min
OnUnitActiveSec=1min
AccuracySec=5s
Persistent=false

[Install]
WantedBy=timers.target

Before deploy, review script for permissions, disk quota, rotation, sensitive data; canary on single host. Disable rollback with systemctl disable --now cpu-snapshot.timer; does not auto-delete collected files, handle per retention policy.

Practical Triage Order

Facing ongoing high CPU: first save timestamp and overall state; use pidstat to find process/thread; parallel confirm business impact; based on usr/sys/soft/steal choose perf, strace, interrupt, or platform path; then correlate deployments and traffic. Facing ended spike: prioritize historical metrics, journal, release records, accounting data. Facing unresponsive host: first protect business via rate limiting, drain, or failover, then preserve scene on remaining healthy replicas.

Any root cause report must separate facts, inferences, and items to verify. Facts are command, log, metric results in explicit time window; inferences are mechanisms they support; verification is expected changes from fix or reproduction experiment. This avoids unactionable conclusions like "CPU high because traffic high".