Diagnosing High Load with Low CPU on Linux: Commands and Tips

1. Common Load Analysis Methods

CPU high, Load high

Use top to find the PID of the process consuming the most CPU.

Use top -Hp PID to find the thread (TID) with the highest CPU usage.

For Java programs, run jstack to print thread stack information.

Use printf %x tid to display the hexadecimal ID of the most CPU‑intensive thread.

CPU low, Load high

The cause is usually an excess of processes waiting for disk I/O, which lengthens the run‑queue while the CPU remains idle.

Run top and check the %wa column for I/O wait time.

Run iostat -d -x -m 1 10 (install with yum install -y sysstat) to view disk I/O statistics.

Run sar -n DEV 1 10 to see network I/O.

Use the following command to locate I/O‑heavy processes:

ps -e -L h o state,cmd | awk '{if($1=="R"||$1=="D"){print $0}}' | sort | uniq -c | sort -k 1nr

2. CPU High, Load High Analysis

Use vmstat to view system‑wide CPU load.

Use top to view per‑process CPU load.

2.1 Using vmstat

Command format: vmstat -n 1 (refreshes every second).

[root@VM-1-14-centos ~]# vmstat -n 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 250304 163472 2154300    0    0     1    16    0    4  1  0 98  0  0
 ... (additional lines omitted for brevity) ...

Key columns:

r : Number of processes waiting for CPU; a high value indicates a long run‑queue.

b : Number of blocked processes.

us : CPU time spent in user space.

sy : CPU time spent in kernel space (system calls, often I/O).

wa : Percentage of CPU time spent waiting for I/O.

id : Percentage of idle CPU time; sustained 0% with high sy suggests CPU shortage.

Common issues and remedies:

If r > 4 and id < 40, the CPU is heavily loaded.

If pi or po are non‑zero, memory is insufficient.

If disk > 0 and b queue > 3, I/O performance is poor.

2.2 Using top for per‑process view

Run top to see CPU, memory, and other resource usage per process. Press P to sort by CPU usage descending.

top - 19:49:59 up 36 days, 23:15, 3 users, load average: 0.11, 0.04, 0.05
Tasks: 133 total, 1 running, 131 sleeping, 0 stopped, 1 zombie
%Cpu(s): 3.1 us, 3.1 sy, 0.0 ni, 93.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
... (process list omitted) ...

The third line shows overall CPU usage; the table below lists each process's consumption.

3. CPU Low, Load High

Problem description : A Linux server without active business processes shows a high load average while CPU usage is low.

Analysis : The high load is caused by many processes waiting for disk I/O, inflating the run‑queue. Typical scenarios include:

Scenario 1: Excessive disk read/write requests leading to I/O wait.

Scenario 2: MySQL queries without indexes or deadlocks causing I/O blockage.

Scenario 3: External storage (e.g., NFS) failures causing prolonged I/O waits.

Remediation :

Understand that load average reflects the length of the task queue; a high value with low CPU indicates I/O bottlenecks.

Check for zombie or uninterruptible (D‑state) processes using ps -axjf or the command shown earlier.

D‑state processes cannot be killed; resolve by restoring the dependent resource or rebooting.

ps -e -L h o state,cmd | awk '{if($1=="R"||$1=="D"){print $0}}' | sort | uniq -c | sort -k 1nr