Diagnosing Linux CPU Spikes: From mpstat to perf Root Cause Analysis

Assess Impact Scope: System-Wide vs Single Core

Monitoring alerts often show average CPU usage across all cores. A 16-core machine with one core at 100% and others idle shows only ~6% average. Use mpstat -P ALL 1 3 to examine per-core utilization. The output reveals three patterns:

All cores uniformly high : Overall compute shortage or global issue (e.g., disk bottleneck queuing all requests).

Single or few cores high : Single-thread bottleneck; proceed to thread-level CPU analysis.

High-low alternating, periodic spikes : Scheduled tasks (cron, batch jobs, GC cycles, log rotation); correlate spike timestamps with crontab and logs.

Simultaneously verify business symptoms: P99 latency degradation, error rate increase, or silent high CPU (e.g., nightly batch). This determines remediation priority and acceptance criteria — whether CPU returns to baseline or business metrics recover.

Decompose CPU Time: user, system, iowait, steal, softirq

vmstat 1 5

provides the next diagnostic layer. Key columns and their implications:

us (user) : High user time points to application code — compute-intensive tasks, regex backtracking, serialization, GC. Next step: drill down by process/thread.

sy (system) : High kernel time suggests syscall intensity (small file I/O, frequent network I/O, container runtime), lock/futex contention, context-switch overhead. Watch cs (context switches/sec); sustained tens of thousands warrants investigation.

wa (iowait) : Tasks waiting for I/O. Crucial misconception : high iowait ≠ disk bottleneck. It means runnable tasks are blocked on I/O completion. Must combine with iostat -x to check queue depth and await.

st (steal) : Virtualization-specific. Host overcommit causing VM to wait for CPU. Sustained >5% requires cloud provider engagement or instance migration; no machine-side fix.

si (softirq) / interrupts : Network card multi-queue not enabled, traffic hitting single softirq core. mpstat shows si column (newer procps).

This layer produces an evidence-hypothesis matrix: observed metric → suspected cause categories. Do not remediate yet; continue drilling.

Identify Process and Thread

With machine-level profile, use top to narrow to processes. RHEL default top shows per-process totals; switch to thread view faster:

top -b -n 1 | head -25
# After identifying suspect PID:
top -H -p 10231 -b -n 1 | head -30

top -H

output includes VCPU column answering two questions: which thread consumes most CPU (sort by %CPU), and whether hotspot stays on same core (correlate with mpstat -P). Thread names provide clues: JVM GC threads like GC Thread #X, G1 Young Remaking; Nginx uses worker processes; Go services vary with GOMAXPROCS.

Supplement with pidstat -u -t -p 10231 1 5 for per-thread user/kernel time split (requires sysstat service running for historical data). If many processes show high CPU, it's resource exhaustion; if few, proceed to scenario-based attribution.

Scenario-Based Attribution

Java Services: Check GC First

High CPU in Java often stems from GC. Use jstat -gcutil <pid> 2000 to observe:

# Example output (every 2 sec)
# S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU FGC FGCT YGC YGCT
# 51200.0 51200.0 0.0 48211.0 524288.0 502130.0 524288.0 380211.0 131072.0 125804.0 10240.0 8192.0 21 142.31 1520 42.10

Interpretation:

FGC (Full GC count) rising steadily with fast-growing FGCT (Full GC time) → frequent Full GC, CPU burned in GC threads (visible in top -H).

YGC (Young GC count) extremely high with short duration → heap too small causing excessive Young GC. Root cause direction: heap sizing, allocation rate (large temporary objects in hot path), or memory leaks preventing Full GC from reclaiming.

Separate GC threads from business threads in top -H; if business threads also high, the code itself has issues beyond GC.

C/C++ and General Processes: perf Sampling

With PID, use perf for user-space call-stack sampling to attribute CPU to functions:

# Sample 30 seconds, generate perf.data
perf record -F 99 -g -p <pid> -- sleep 30

# Interactive hotspot view
perf top -p <pid>
# Example output:
# 28.4% app libstdc++.so.6 [.] std::__detail::_Map_base::find
# 17.1% app app [.] request_parser::parse_header

perf record

requires perf_event_paranoid permission (container may need --cap-add SYS_ADMIN or kernel param adjustment). First verify symbols visible with perf top -p <pid>; missing debug info reduces hotspots to offsets. With -g (call graph), perf report shows callers of hot functions, revealing "which code path burns CPU".

High Kernel (sy) Time: Context Switches and Syscalls

When vmstat shows sustained cs > tens of thousands/sec with significant sy, suspect lock contention or futex overhead:

# perf kernel symbols (needs root)
perf top -p <pid> | head
# Example: futex_wait 20%+ → many threads waiting on same lock

High futex indicates severe thread synchronization (busy-wait, spinlocks, coarse-grained locks). Next: identify the specific lock via perf record -g combined with source code lock locations, or GDB attachment to observe lock holder. Another sy spike source: syscall storms — synchronous fsync logging, massive small socket I/O, excessive inotify watches. perf top showing vfs_*, socket_* symbols distinguishes this.

High iowait: iostat Queue Analysis, Not Just Numbers

When vmstat shows high wa, confirm actual disk queuing with iostat -x 1 3:

# Example output
# Device rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgsz await svctm %util
# sdb 0.00 0.00 120 800 6000 180000 450.0 1.20 9.80 100.0

Judge by await (avg I/O wait ms) and %util : - Low await (single-digit) and %util < 100% → I/O not slow; wa may be sporadic waits, disk not root cause. - Await significantly above device baseline (HDD 5-10ms, SSD sub-ms to ms) and %util near 100% → disk queuing is primary issue. Then identify heavy readers/writers via iotop (if available) or pidstat -d. - Hidden pattern: disk healthy but kernel running md / lvm /RAID rebuild or backup software consuming I/O bandwidth; appears in iostat per device.

Softirq and Single-Core Network Interrupts

When a core shows sustained high si (softirq) on network-intensive workload, check interrupt distribution: watch -n 5 'grep eth0 /proc/interrupts | head' If one RX queue interrupt count grows while others static, NIC multi-queue/RSS not effective. Verify queue count via cat /sys/class/net/eth0/queues/, check irqbalance running, and ensure no manual IRQ affinity pinning all interrupts to one core.

Containerized Services: cgroup Throttling as Separate Category

Containers introduce a "high CPU" invisible to machine-level tools: CPU usage below limit but CFS throttling causes latency spikes. Inside container (or host via container cgroup path):

# cgroup v2 example
# nr_throttled increasing = throttled scheduling periods rising
grep -E 'nr_throttled|throttled_usec' /sys/fs/cgroup/<container>/cpu.stat

nr_throttled

growing while container CPU usage < 100% limit indicates limit set too low: burst traffic exceeds CFS quota (quota/period time slice exhausted), causing periodic "freeze". In Kubernetes, check spec.containers[].resources.limits.cpu. Remediation: increase limit or optimize burst — note cgroup v1 vs v2 path differences (v1 at /sys/fs/cgroup/cpu,cpuacct/<container>/cpu.stat); first run stat /sys/fs/cgroup to determine version.

Root Cause Classification, Remediation, and Rollback

After identifying function or config, causes typically fall into these categories with distinct remediation/rollback paths:

Code defect (algorithmic complexity, regex catastrophic backtracking, infinite loop): hotfix or version rollback. Pre-rollback verify data-layer compatibility; post-rollback validate metrics return to pre-change baseline for at least one full traffic cycle.

GC misconfiguration : Heap size or GC algorithm change is a change operation. Test new params in staging under load; keep old config file; rollback = restore old params + restart.

Cron job peak collision : Scheduled task clashes with business peak. Reschedule or add nice /CPU limits. No rollback; only revert schedule.

Steal / host overcommit : Machine-side unsolvable. Contact cloud provider or migrate; ensure snapshot or rollback-capable release channel before migration.

cgroup throttling : Increase K8s CPU limit (change). Canary: adjust one replica first, observe; rollback = restore old limit.

I/O bottleneck : Device upgrade, write pattern change (batch writes, O_DIRECT) not immediate. Temporary mitigation (rate limiting, degrade write-heavy features) must define stop conditions and recovery steps.

For any remediation, preserve "current state" evidence: capture mpstat, vmstat, jstat samples (store under /var/log/, watch disk space). After action, collect same commands again; measure root cause elimination with identical ruler.

Verification and Postmortem

Verification goes beyond CPU returning to baseline:

CPU metrics (system-wide, per-core, suspect process/thread) stable for at least one full business cycle (covering peak hours).

Business KPIs (P99, error rate, throughput) restored to pre-incident baseline — if CPU drops but P99 doesn't recover, high CPU was symptom, true business impact unresolved.

If config/params changed, confirm rollback artifacts exist and rollback procedure executable.

Postmortem checklist:

Evidence chain complete? (Alert screenshots, mpstat/vmstat samples, perf reports, timestamps archived.)

Monitoring gaps: Can alerts fire on "single core saturated" (not just system average)? Are process-level CPU, jstat GC metrics, container nr_throttled ingested?

Periodic spikes: Added silence windows or threshold differentiation for known time windows?

Key Diagnostic Principles to Remember

Alert "CPU usage" is almost always system average. First split per-core ( mpstat -P ALL); pattern (all-core/single-core/periodic) dictates suspicion direction.

Second layer: vmstat user/system/iowait/steal + cs buckets problem into "app code, kernel overhead, I/O, host overcommit, lock contention" — each with fixed toolset (perf, jstat, iostat, /proc/interrupts).

High iowait ≠ "disk broken". Must check iostat -x await and %util against that disk's historical baseline.

Containers separate bucket: CFS throttling ( nr_throttled rising while CPU < limit) is a "high latency" source invisible to machine-level analysis.

Every remediation is a change: pre-change evidence capture (same commands, two samples), post-change dual acceptance (CPU metrics + business metrics), rollback path documented before execution.