Linux High CPU: Real vs Fake vs Stolen — 6-Step Systematic Troubleshooting Guide

Three Types of High CPU

The article opens by classifying high CPU into three categories, each requiring different handling:

Real high : A process genuinely consumes CPU cycles (high us or sy, low id). Causes include infinite loops, regex backtracking, crypto ops, or GC thrashing.

Fake high : The CPU isn't actually saturated; the measurement is misleading. Four common pitfalls: top 's first screen shows boot-time average, not current; containers see host CPU not their limit; monitoring intervals (e.g., 5 min) smooth out spikes; multi-core summary hides a single saturated core.

Stolen high : CPU time is taken by the hypervisor ( steal), container throttling ( nr_throttled), or interrupts/softirqs ( si > 30%). No culprit process appears because your tasks are queued.

Six-Step Diagnostic Path

The author prescribes a fixed workflow with concrete outputs at each step:

Step 0 – Verify reality : Use top -b -n2, mpstat -P ALL 1 3, cat /proc/pressure/cpu to get current (not average) utilization and PSI stall metrics.

Step 1 – System snapshot (60 sec) : Run a script capturing vmstat, mpstat, pidstat, sar, /proc/interrupts, /proc/softirqs, dmesg. Outputs are saved for offline analysis.

Step 2 – Identify the process : ps --sort=-pcpu, systemd-cgtop (cgroup-aware), pidstat -u -w. If no user process shows high CPU, check kernel threads ( kworker, ksoftirqd), interrupts, or steal/throttling.

Step 3 – Identify the thread : top -H -p PID, ps -L -p PID, pidstat -t -p PID. The wchan column reveals if a thread is running ( -) or waiting (e.g., futex_wait_queue). Map TID to language-specific names (Java jstack + nid, MySQL performance_schema.threads).

Step 4 – Locate the code : Prefer language-native profilers: Java async-profiler, Go pprof, Python py-spy, Node 0x. Fallback to perf (requires perf_event_paranoid=1 and debuginfo packages) or bpftrace for kernel/user stacks without symbols. strace -c only for syscall frequency, with timeout to limit overhead.

Step 5 – Conclude by pattern : A decision table maps observed patterns (high us, high sy, high wa, high si, high st, high nr_throttled, high context switches, single-core saturation) to root causes and verification commands.

Key Tools & Metrics Explained

CPU Time Fields

Detailed breakdown of top / mpstat columns: us (user), sy (system), ni (nice), id (idle), wa (iowait), hi / si (hard/soft irq), st (steal). Rules of thumb: sy > 40% of us+sy warrants investigation; sy > us almost certainly indicates a problem.

PSI (Pressure Stall Information)

From /proc/pressure/cpu, some avg10 > 20% means noticeable queuing; > 50% near collapse. Requires kernel 4.20+ with CONFIG_PSI=y.

cgroup v2 Throttling

Read cpu.max (quota/period) and cpu.stat ( nr_throttled, throttled_usec). nr_throttled growing = container CPU limit too low. Kubernetes metric: container_cpu_cfs_throttled_periods_total.

Hyper-threading & Single-Core Bottlenecks

lscpu

shows threads per core. Logical cores ≠ physical throughput; a single-threaded app on one core at 100% caps its throughput regardless of total cores.

Six Real-World Scenarios

1. Regex Catastrophic Backtracking

Content moderation service CPU 15% → 95%, QPS flat. top -H showed 4 threads at ~25% each. jstack revealed all stuck in Pattern$Loop.match. Root cause: nested quantifier (a+)+b on 200KB input. Fix: disable rule, rewrite regex to a+b, add length guard and timeout.

2. JVM Full GC Storm

Order service CPU spikes to 300% every 40 min for 20 sec. jstat -gcutil showed old gen (O) at 99%, FGC rising every 5 sec. Cause: unbounded ConcurrentHashMap cache filling heap. Fix: switch to Caffeine with maximumSize and expireAfterWrite; add heap usage alert.

3. Lock Contention Driving sy

Payment callback 16 cores, 60% CPU ( us 25%, sy 35%), throughput 1/3 of expected. pidstat -w showed 48k voluntary context switches/sec. perf top hotspots: futex_wait_queue_me, __pv_queued_spin_lock_slowpath. Root cause: global synchronized counter. Fix: replace with LongAdder (striped) or async batching. Result: cswch/s 48k → 3k, sy 35% → 8%.

4. SoftIRQ Single-Core Saturation

Gateway 8 cores, 30% overall CPU but P99 latency 20ms → 400ms. mpstat -P ALL showed CPU0 %soft 96%. /proc/net/softnet_stat CPU0 dropped packets climbing. Cause: 8-queue NIC but irqbalance disabled, all interrupts on CPU0. Fix: enable irqbalance (or manual smp_affinity), verify interrupt spread. Result: softirq distributed, drops stopped, P99 back to 25ms.

5. Cron Job Collision

DB server CPU 20% → 100% daily at 02:00 for 8 min. crontab audit found three jobs at 0 2 * * * (backup, stats, log archive) all full-table scans. Fix: stagger to 02:00, 02:20, 02:40 and add flock -n mutex in scripts. Next day peak CPU 55%, replication lag normal.

6. Container CPU Limit Throttling ("Fake Idle")

K8s pod CPU limit 2 cores, usage shows 45% but P99 jitter. cpu.stat revealed nr_throttled / nr_periods = 23.8%, throttled_usec 2061 sec. Cause: burst workload (0.5 core steady, spikes to 3 cores for 200ms) clipped by 2-core limit. Fix: raise limit to 4 cores short-term; tune batch size mid-term; long-term consider removing CPU limit (only request) with node isolation.

Remediation Actions & Risks

kill -TERM

then kill -KILL – graceful then force; ensure multi-instance, health-check pass, restart policy. renice -n 19 – lowers priority but doesn't reduce total CPU; ineffective under saturation; risk of starvation. taskset -pc 0-3 – pins process to cores; immediate effect, lost on restart; wrong pinning can worsen contention. systemctl set-property CPUQuota=200% / K8s kubectl set resources – enforces limit via throttling; may trade CPU high for latency high; verify with cpu.stat first. cpulimit – uses SIGSTOP/SIGCONT; not recommended for production latency-sensitive services.

Verification & Postmortem

Validation checklist: CPU below baseline+10% ( mpstat 60s), PSI avg10 < 5, business P99/error-rate normal, no new errors, headroom < 70% at peak. Observe at least one full business cycle (24h). Postmortem must answer: detection source, timing cause, monitoring gap, secondary effects, prevention (script, alert, doc).