Understanding Linux Load: Calculation, Tools, and Advanced Monitoring

1. Accurate Meaning of Linux Load

In daily operations we often encounter high Linux load, but there is no systematic article explaining how the load is calculated. This article fully explains the calculation principle of Linux load and methods to investigate influencing factors.

Typical ways to obtain the load are the commands top, w, and uptime, which all read the values from /proc/loadavg. You can view the file directly with:

The commands belong to the procps-ng (or older procps) package. The /proc directory is a pseudo‑filesystem that provides an interface to kernel data structures.

The first three fields of /proc/loadavg are the average number of jobs (processes or threads) over the last 1, 5, and 15 minutes. A job includes tasks in state R (running) or D (uninterruptible I/O).

/proc/loadavg’s first three numbers represent load1 , load5 , and load15 .

The load value is the average number of jobs during the corresponding interval. The term “job” is loosely used; a more accurate term is “task” (kernel) or “thread” (user space).

Only tasks in states R and D are counted; other states are ignored.

2. Decomposing Linux Load with the load2process Script

Understanding the kernel code that computes the load is difficult, so the load2process script provides a practical way to break down the load into per‑process contributions.

The script includes two tools: load2process and load2pid. It uses ps with the following options: -e: show all processes. -L: list every thread of each process on a separate line. h: hide the header line. o state,ucmd: output only the thread state and command name. o state,pid,cmd: output state, PID, and full command line.

Running the script on a busy machine yields a table where the first column sums to a value close to load1. The rows with state R identify the threads that contribute most to the load.

Additional usages of load2process are shown below, and when the third column contains names like java , python , or php , the load2pid tool can further decompose the load per PID.

3. More Sensitive load5s and Load Prediction

To obtain a load metric more responsive than load1, a kernel module load5s provides a 5‑second average. After installing load5s.ko, the value appears in /proc and changes only every 5 seconds.

When load1 is high, a low load5s suggests the system is already recovering. The load5s value also helps reveal the calculation logic of load1, load5, and load15. A sample prediction script load_predict.sh demonstrates that the predicted values match the actual ones within 0.01.

4. Kernel Code Analysis of Load

Using kernel version 3.10.0 as an example, the /proc/loadavg pseudo‑file is generated by the macro LOAD_INT(avnrun[0]) and LOAD_FRAC(avnrun[0]), which effectively compute avnrun[0] / 2048 with two decimal places.

The function get_avenrun in kernel/sched/core.c returns the global avenrun array, which is updated every 5001 ms by calc_load. The three load averages differ only by the exponential decay constant passed to calc_load (1884 for 1 min, 2014 for 5 min, 2037 for 15 min).

The active task count comes from calc_load_tasks, which aggregates nr_running (R state) and nr_uninterruptible (D state) from each CPU run‑queue.

5. Disassembly of the Load‑Related Kernel Code

To understand load5s implementation, the kernel binary ( vmlinuz) must be decompressed and disassembled with objdump -d -M att. Using the debug package ( kernel‑debug) provides symbol information, making the analysis easier.

6. Kprobe Implementation of load5s

load5s

uses the kprobe mechanism. By locating the address of the global calc_load_tasks structure via kallsyms_lookup_name, the probe can read the sum of R and D tasks without modifying kernel code. The correct offset (e.g., 0x23d) is determined by matching assembly instructions.

The final load5s.ko module can be built with a standard make process.

7. Mathematical Basis of Load Calculation

The kernel uses an exponential decay formula: load = load * exp(-x/period) + active * (1 - exp(-x/period)), where x is the sampling interval (5 s) and period is 60, 300, or 900 s for the three averages. This yields the constants 1884, 2014, and 2037 used in the code.

8. Lightweight Monitoring Alternatives

While load5s requires a kernel module, the load2process -s option provides a real‑time sum of R + D threads. A monitoring script check_load_process wraps this tool and returns JSON with two thresholds: load_threshold (default 2 × CPU count) and thread_threshold (default 0.4 × CPU count).

9. Different Impacts of R and D States on Load

R‑state dominance makes the system feel sluggish; if R threads exceed twice the CPU core count, severe contention occurs. D‑state dominance can produce extremely high load numbers (e.g., >10 000) while the system remains responsive, because D threads are sleeping on I/O.

The /proc/loadavg file also provides nr_running (the value before the slash), which matches the runq‑sz column of sar. A high load with low nr_running indicates D‑state pressure.

10. Diagnosing High Linux Load

Four typical scenarios are identified:

Single R‑state thread causing high load (e.g., a CPU‑bound process).

Multiple R‑state threads contributing collectively.

Single D‑state thread (often I/O wait or kernel mutex).

Multiple D‑state threads (e.g., disk problems).

Understanding whether R or D threads dominate, and using tools like top, wchan, and stack, helps pinpoint the root cause.

In summary, a deep knowledge of kernel load calculation, the distinction between R and D states, and the use of scripts such as load2process, load2pid, and the load5s module enables precise monitoring and troubleshooting of Linux system load.