Step‑by‑Step Guide to Diagnose 100 % CPU on a Linux Server

1. Define What “CPU 100 %” Means

CPU 100 % can be reported by different sources (monitoring platforms, top, container panels, cloud consoles) with inconsistent denominators. For example, on an 8‑core machine a single‑threaded process may show ~100 % in top while the overall average is only ~12.5 %.

Check the monitoring metric (e.g., node_cpu_seconds_total in Prometheus) and understand whether it reflects a single logical CPU or the whole machine.

Record the alert time, duration, aggregation target, and sampling interval. Capture hostname, service instance, alert start time, whether it is still ongoing, the query used, and recent deployment or configuration changes.

2. Protect the Environment and Collect Information

Log into the correct host and verify the system time. Run low‑impact commands first:

hostnamectl
date --iso-8601=seconds
uptime

uptime

shows load average, which is not the same as CPU usage; high load may be caused by CPU contention or I/O wait.

Identify logical CPU count and topology:

lscpu
nproc

Perform short‑duration, low‑impact sampling rather than a single snapshot:

LC_ALL=C top -b -d 1 -n 10 > /tmp/top-$(date +%Y%m%d-%H%M%S).txt
vmstat 1 10
mpstat -P ALL 1 10
pidstat -u -r -d -w 1 10

When disk space in /tmp is limited, check it first:

df -h /tmp
df -i /tmp

Avoid deleting historical logs during sampling; if log cleanup is required, archive and verify backups first.

3. Use System‑Level Metrics for Initial Diagnosis

3.1 Examine CPU Time Distribution with mpstat

%usr

– time spent in user‑mode code. %sys – time spent in kernel‑mode. %iowait – CPU idle while I/O requests are pending. %irq / %soft – hardware and software interrupt time. %steal – time a virtual CPU waited for the host. %idle – idle time.

Interpretation examples:

If %usr stays high, investigate business processes, request volume, and code hotspots.

If %sys or %soft is high, look at system calls, lock contention, network packet processing, or frequent context switches.

If %steal is high on a VM, correlate with cloud‑host metrics and consider migration.

3.2 Look at Run Queue and Context Switches with vmstat

vmstat 1 10

Key fields: r – runnable tasks; consistently above the number of logical CPUs indicates queue contention. b – tasks in uninterruptible sleep (I/O wait). cs – context switches per second; compare with historical baseline. us, sy, id, wa, st – user, system, idle, I/O wait, and steal percentages.

3.3 Review Historical Data with sar

sar -u -s 14:00:00 -e 14:30:00
sar -q -s 14:00:00 -e 14:30:00

If sysstat was not enabled, fall back to monitoring platforms, application logs, and change records.

4. From Process to Thread

List processes by CPU usage while preserving full arguments:

ps -eo pid,ppid,user,stat,psr,pcpu,pmem,etime,lstart,comm,args --sort=-pcpu | head -n 30

Important columns: PID / PPID – process hierarchy. PSR – last logical CPU the process ran on (not a binding). STAT – R (running/runnable), D (uninterruptible sleep), Z (zombie). ETIME / LSTART – helps align with the fault window. ARGS – full command line (sanitize sensitive parameters before sharing).

For a confirmed PID ( $PID), drill down:

PID=12345
ps -p "$PID" -o pid,ppid,user,stat,psr,pcpu,pmem,etime,lstart,cmd
readlink -f "/proc/$PID/exe"
tr '\0' ' ' < "/proc/$PID/cmdline"
printf '
'
cat "/proc/$PID/status"
cat "/proc/$PID/limits"

For multithreaded programs, inspect threads:

top -H -p "$PID"
pidstat -t -u -p "$PID" 1 10
ps -L -p "$PID" -o pid,tid,psr,stat,pcpu,comm --sort=-pcpu

Map Linux thread IDs to Java thread IDs when needed:

TID=12367
printf '0x%x
' "$TID"

Collect Java thread dumps (if JDK tools are available and permissions allow):

jcmd "$PID" Thread.print -l > "/tmp/jcmd-thread-$PID-$(date +%Y%m%d-%H%M%S).txt"

Do not use kill -3 blindly; the dump location depends on the JVM’s stderr configuration.

5. Build an Evidence Chain for Different CPU Types

5.1 User‑Mode CPU High

Typical causes: compute‑intensive logic, infinite loops, traffic surge, serialization, regex backtracking, encryption, GC. Verify request volume, per‑request latency, thread hotspots, and deployment time. pidstat -u -r -w -p "$PID" 1 10 For function‑level hotspots, use perf (subject to kernel.perf_event_paranoid, debug symbols, container permissions):

perf top -p "$PID"
perf record -F 99 -g -p "$PID" -- sleep 30
perf report

5.2 Kernel‑Mode CPU High

When %sys or %soft is high, check system‑call rate, context switches, network packet rate, and kernel logs.

pidstat -w -p ALL 1 10
cat /proc/softirqs
cat /proc/interrupts
journalctl -k --since '-30 min' --no-pager

For suspected syscall storms, sample a short period:

timeout 15s strace -f -c -p "$PID"

Be aware that strace adds overhead, especially for high‑frequency syscalls.

5.3 I/O Wait Misinterpreted as CPU

If %iowait, b, and block device latency rise together, the bottleneck is likely storage.

iostat -xz 1 10
pidstat -d -p ALL 1 10

Check r/s, w/s, r_await, w_await, and %util (interpret according to sysstat version and device type).

5.4 VM Steal High

When %steal stays high without a corresponding high‑CPU process, the guest is waiting for host CPU. Correlate with mpstat / sar steal time, cloud‑host metrics, and business latency. Fixes may involve migrating the instance or adjusting host resources.

6. Containers and Kubernetes

Confirm the process belongs to a cgroup:

cat "/proc/$PID/cgroup"
systemd-cgls
systemd-cgtop

For cgroup v2, read cpu.stat, cpu.max, and cpu.pressure from the appropriate path.

CGROUP_PATH=/sys/fs/cgroup/example.slice/example.service
cat "$CGROUP_PATH/cpu.stat"
cat "$CGROUP_PATH/cpu.max"
cat "$CGROUP_PATH/cpu.pressure"

In Kubernetes, identify the namespace and pod, then use the metrics server and Prometheus exporters:

NAMESPACE=production
POD=example-pod
kubectl -n "$NAMESPACE" top pod "$POD" --containers
kubectl -n "$NAMESPACE" describe pod "$POD"
kubectl -n "$NAMESPACE" get pod "$POD" -o yaml

Check container CPU requests/limits and throttling metrics such as container_cpu_usage_seconds_total, container_cpu_cfs_throttled_seconds_total, and container_cpu_cfs_throttled_periods_total.

7. Form a Reproducible Root‑Cause Conclusion

A valid conclusion includes four evidence categories:

Time evidence – CPU anomaly aligns with deployment, traffic spike, or dependent failure.

Resource evidence – mpstat, vmstat, pidstat or monitoring show where CPU time is spent.

Object evidence – the same process/thread/container consistently appears as the hotspot.

Negative evidence – storage wait, VM steal, CPU quota throttling, or metric mis‑configuration have been ruled out.

Example: sustained high %usr with a running queue above CPU count, a JVM process consuming most user CPU, and thread dumps pointing to the same code path, while request volume, I/O wait, and steal remain unchanged.

8. Low‑Risk Remediation Steps

If abnormal traffic caused the spike, apply rate‑limiting, degradation, or isolate the entry point.

If a background job is responsible, pause or reduce concurrency instead of killing all workers.

If container CPU limit is too low, increase it after confirming node capacity and gray‑deploy.

If a new code version introduced the hotspot, roll back to a verified version while preserving samples.

If a single instance is out of control, detach traffic first, then consider graceful stop or restart.

Restarting a service is high‑risk; before doing so, verify impact scope, health of other instances, backup diagnostics, use graceful shutdown, and have a rollback plan.

9. Verification After Fix

Observe a full business‑cycle window. Short‑term drops may be due to traffic shift or cold start.

mpstat -P ALL 1 10
vmstat 1 10
pidstat -u -r -d -w 1 10

Check that %usr, %sys, %iowait, and %steal return to baseline, run‑queue and context‑switch rates fall, the target process/thread hotspot disappears, request volume recovers, and error/latency metrics normalize. Also verify no new pressure appears on memory, disk, or downstream services.

10. Post‑Mortem and Long‑Term Governance

Monitor user‑mode, kernel‑mode, I/O wait, steal, run‑queue, and container throttling separately.

Retain sufficient process‑level or container‑level history and keep clocks synchronized across hosts, containers, and deployment systems.

Audit scheduled tasks, batch jobs, and thread‑pool configurations to prevent unbounded concurrency.

Prepare verifiable rate‑limit, degradation, traffic‑isolation, graceful shutdown, and rollback procedures.

Establish CPU baselines in load‑test or pre‑release environments and record normal request, latency, and throughput metrics.

Enhance high‑CPU alerts with duration and business‑metric correlation to reduce false positives.

CPU 100 % is an entry point, not a root cause. A reliable investigation starts with confirming the measurement window, determining the type of CPU time, tracing from system to process to thread, and closing the loop with logs, metrics, sampling hotspots, and change records.