How to Quickly Identify High‑CPU Processes on a Linux Server with a One‑Minute Command Checklist

CPU spikes are one of the most common performance problems on Linux servers, often manifesting as alerts over 90% usage, slow responses, time‑outs, or sluggish SSH sessions. The article presents a structured, three‑phase troubleshooting workflow that moves from a quick global view to detailed per‑process analysis.

Phase 1 – 60‑second global scan

After logging in, run a set of commands to determine whether the bottleneck is CPU, memory, I/O or network: uptime – check the load average against the number of logical CPUs (shown by nproc or lscpu). top – examine the %Cpu(s) line; high us (>70%) points to CPU‑bound work, high sy (>30%) suggests heavy system calls or driver issues, high wa (>20%) indicates I/O wait, and low id (<10%) means the CPU is saturated. vmstat 1 5 – focus on the first four columns and the CPU columns. r > CPU count signals a saturated run queue; b > 0 shows blocked I/O or locks; cs > 50 000 indicates frequent context switches; wa > 20% confirms I/O pressure. mpstat -P ALL 1 3 – look for cores where %usr or %sys is much higher than the rest, which may reveal CPU pinning or a single‑thread bottleneck.

Based on these metrics, use the decision matrix to choose the next step (e.g., high us + high r → go to Phase 2; high sy + high cs → investigate lock contention with strace / perf).

Phase 2 – Locate the offending process and thread

pidstat -u 1 5 | sort -k8 -nr | head -15

– list processes by CPU usage; the %wait column shows how much time a process spends waiting for the scheduler. top -H -p <PID> or

ps -p <PID> -To pid,tid,%cpu,cmd --sort=-%cpu | head -10

– drill down to the thread level and note the TID of the hottest thread.

Convert the TID to hexadecimal (e.g., printf "%x\n" 6789) for use in thread dumps ( jstack, pstack, or /proc/<PID>/task/<TID>/stack). perf top -p <PID> -g – see which functions consume CPU, with call‑graph information. sudo perf record -F 99 -p <PID> -g --sleep 30 followed by sudo perf script -i perf.data > out.perf and the FlameGraph scripts from https://github.com/brendangregg/FlameGraph to generate a flame‑graph ( ./flamegraph.pl out.folded > cpu.svg).

If %sy is high, run sudo strace -c -p <PID> to see the distribution of system‑call time (e.g., frequent futex indicates lock contention, epoll_wait shows normal I/O wait, write/read points to heavy disk activity).

Phase 3 – Classify root cause and deep‑dive

The article groups typical causes into eight categories, each with characteristic symptoms and diagnostic commands:

CPU‑bound business logic – %usr > 70%, load > CPU count; use perf top / perf record to locate hot functions.

Infinite loop / recursion – a single process at 100% CPU with no I/O; strace -c and thread dumps reveal the looping code.

Lock contention – %sy > 30% and cs > 50 000; high futex frequency in strace -e futex or perf record -e sched:sched_stat_runtime helps isolate the lock.

Excessive context switches – high load with high cs (>80 000) but low CPU usage; pidstat -w distinguishes voluntary vs. involuntary switches.

Interrupt overload – unusually high %si (soft) or %hi (hard); inspect /proc/interrupts and /proc/softirqs.

vCPU steal on cloud VMs – %st > 5% and low idle; monitor with mpstat -P ALL and raise a ticket with the cloud provider.

Production‑environment considerations

Limit perf record -F to 99–997 Hz to avoid self‑induced overhead.

Include kernel stacks ( -k 1) or the flame‑graph will be filled with [unknown].

Use strace -p only during low‑traffic windows; it can slow a high‑concurrency process 5–10×.

Avoid blind kill of high‑CPU processes unless you are certain they are not serving business traffic.

Check for CPU pinning with taskset or missing irqbalance when a single core shows 100% usage.

If %st remains high, local tuning is ineffective – contact the cloud vendor.

Always capture a snapshot before changing anything (date, uptime, top -bn1, vmstat 1 5 into a log file).

Summary

Start with a 60‑second global scan (uptime, top, vmstat, mpstat) to classify the bottleneck.

Lock in the suspect process using pidstat, perf top, or strace, then drill to the thread and function level.

Apply the targeted remedy based on the identified root cause – algorithm optimisation, lock reduction, scaling resources, or configuration tweaks.

Blind restarts or adding hardware only mask the symptom; systematic diagnosis and targeted remediation are the only reliable ways to eliminate recurring CPU spikes.