Demystifying Linux Load: Calculation, Tools, and Advanced Monitoring

The article, originally published on the "Alibaba Intelligent Operations" WeChat account, offers a comprehensive guide to understanding, measuring, and troubleshooting Linux load values.

1. Linux Load: Meaning and Sources

In daily operations, high Linux load is common, yet the calculation method is rarely documented. Load values are obtained via commands such as top, w, and uptime, which read the three numbers from /proc/loadavg. The three fields represent the average number of jobs (running or uninterruptible) over the past 1, 5, and 15 minutes.

The /proc pseudo‑filesystem provides this data; you can view it directly with:

cat /proc/loadavg

Load Fields Explained

The first three numbers are load1, load5, and load15.

Each value is the average number of jobs (threads in state R or D) over the corresponding interval.

Only tasks in states R (running) and D (uninterruptible I/O wait) are counted.

2. Decomposing Load with load2process and load2pid

The script load2process (and its companion load2pid) breaks down the load into per‑process contributions. It uses ps -e -L -o state,ucmd and ps -e -L -o state,pid,cmd to list threads and their states. -e: show all processes. -L: expand each process to show all its threads. -h: hide the header line. -o state,ucmd: output thread state and command name. -o state,pid,cmd: output thread state, PID, and full command.

Running the script on a busy machine yields a table where the sum of the first column approximates load1. The dominant R‑state thread (e.g., tasker_1) often accounts for the majority of the load.

3. Introducing the More Sensitive load5s Metric

To obtain a 5‑second granularity, a kernel module load5s.ko can be built and installed. After loading, /proc/load5s provides a value that updates every 5 seconds, offering a finer‑grained view than load1.

Running the provided load_predict.sh script shows that the predicted load values match the actual load1, load5, and load15 with negligible error.

4. Kernel‑Level Load Calculation

The kernel computes load in kernel/sched/core.c. The macro LOAD_INT(avnrun[0]) extracts the integer part, while LOAD_FRAC(avnrun[0]) extracts the fractional part (scaled by 2048). The function get_avenrun() returns the three load values from the global avenrun array.

Every 5001 ms, calc_global_load() calls calc_load() with different exponential decay parameters (1884 for 1 min, 2014 for 5 min, 2037 for 15 min). The underlying formula uses an exponential decay based on the natural constant e.

5. Kprobe‑Based load5s Implementation

The module uses a kprobe to hook the kernel variable calc_load_tasks, which aggregates nr_running (R‑state) and nr_uninterruptible (D‑state) thread counts. By locating the appropriate offset (e.g., 0xffffffff810c2a56) and registering the probe, the module records the combined count every 5 seconds.

6. Practical Monitoring and Troubleshooting

For production use, the check_load_process script wraps load2process and outputs a JSON report. It accepts optional thresholds load_threshold (default 2) and thread_threshold (default 0.4) to filter out low‑impact threads.

Distinguishing R‑state and D‑state contributions is crucial: a high R‑state count indicates CPU contention, while a high D‑state count may reflect I/O wait without severe CPU impact.

By correlating /proc/loadavg, nr_running, and the script outputs, operators can classify load spikes into four scenarios: single‑thread R‑high, multi‑thread R‑high, single‑thread D‑high, and multi‑thread D‑high, each requiring different remediation strategies.

7. Summary

Understanding the kernel’s load calculation, using fine‑grained tools like load5s, and applying scripted decomposition enable precise detection of performance bottlenecks and informed decision‑making in Linux system operations.