Skip to content
Master Linux Load Average Troubleshooting: From Concepts to Real-World Fixes
Operations 63 min read

Master Linux Load Average Troubleshooting: From Concepts to Real-World Fixes

This comprehensive guide covers Linux Load Average fundamentals, diagnostic commands (top, iostat, vmstat, strace), step-by-step troubleshooting for CPU, memory, disk I/O, network, and process issues, plus remediation tactics, verification methods, and production safety practices.

Raymond Ops
Raymond Ops
Raymond Ops

1. Problem Background

One of the most common operational scenarios in production environments is a sudden spike in server load, causing slow or unavailable business responses. Monitoring platforms frequently alert on high Load Average, and SSH logins become sluggish. Quickly identifying the root cause and taking effective measures to restore service tests an engineer's depth of understanding of the Linux resource model and practical experience.

High Load Average does not necessarily mean high CPU usage or insufficient memory. Real causes may include disk I/O bottlenecks, network traffic surges, zombie process accumulation, excessive soft interrupts, kernel task hangs, and more. Therefore, troubleshooting load issues requires a complete observation chain and judgment logic to avoid blindly restarting services or tuning parameters by guesswork.

This article systematically organizes Linux load core concepts, troubleshooting paths, tool usage, command interpretation, common root causes, and remediation measures from a practical perspective, aiming to form a complete troubleshooting loop.

2. Applicable Scenarios

The content applies to the following typical scenarios:

Monitoring alerts show server Load Average persistently above normal levels

Users report slow or timed-out business responses

SSH login to the server is noticeably laggy

Application logs show large numbers of slow requests or queue backlogs

Database queries slow down, connection counts surge

Container node resource usage is abnormal

Cron jobs or batch processing cause system resource contention

Performance degrades after system upgrades, scaling, or migrations

Whether physical machines, virtual machines, or cloud instances, as long as they run the Linux kernel, the troubleshooting approaches and commands in this article apply.

3. Core Knowledge Points

3.1 What is Load Average

Load Average is a Linux metric measuring system busyness, representing the average number of processes in runnable (R state) and uninterruptible sleep (D state) over a specific time interval.

Typically three values are shown, representing the 1-minute, 5-minute, and 15-minute average loads.

uptime
# Output example:
# 14:23:45 up 10 days, 3:15, 2 users, load average: 2.35, 1.80, 1.50

The values 2.35, 1.80, 1.50 are the Load Average.

R State Processes

Processes in Running or Runnable state, either using CPU or waiting for CPU scheduling.

D State Processes

Processes in uninterruptible sleep (Uninterruptible Sleep), usually waiting for I/O operations to complete, such as disk reads/writes or NFS mount hangs. These processes cannot be interrupted by signals or killed; they can only wait for I/O completion or a system reboot.

3.2 How High is High for Load Average

There is no absolute standard; judgment must combine CPU core count.

General rules:

Load Average close to or less than CPU core count: system normal

Load Average 1.5x CPU core count or more: needs attention

Load Average 2x CPU core count or more: severe congestion

For example, a 4-core server:

Load Average below 3: normal

Load Average between 4-6: needs investigation

Load Average above 8: severe congestion

These are empirical values; business characteristics and historical baselines must also be considered.

3.3 Possible Causes of High Load Average

Excessive CPU-intensive tasks

Disk I/O bottlenecks

High network I/O

Insufficient memory triggering swap

Excessive soft interrupts

Zombie process accumulation

Kernel task hangs

NFS, CIFS, or other remote filesystem mount anomalies

Driver or hardware failures

3.4 Troubleshooting Principles

Overall first, then local: check system-wide metrics first, then specific processes

Observe first, then act: gather sufficient information before deciding

Eliminate obvious issues first: prioritize checking CPU, memory, disk, network

Non-intrusive first, then intrusive: prefer read-only commands, avoid immediate service restarts

Record the scene: save key command outputs for post-mortem analysis

4. Overall Troubleshooting Approach

Recommended flow for troubleshooting high Load Average:

Step 1: Confirm load is truly abnormal
Step 2: Check CPU usage
Step 3: Check memory usage
Step 4: Check disk I/O
Step 5: Check network traffic
Step 6: Check process state distribution
Step 7: Identify specific processes
Step 8: Analyze process behavior
Step 9: Apply remediation measures
Step 10: Verify effectiveness
Step 11: Post-mortem summary

5. Practical Steps

Step 1: Confirm Load is Abnormal

Purpose

Avoid misjudgment; first confirm Load Average indeed exceeds normal range.

Command

uptime

Expected Output

14:30:12 up 10 days, 3:21, 2 users, load average: 8.56, 7.23, 6.10

Abnormal Indicators

1-minute load far higher than 5-minute and 15-minute: burst issue

All three load values high and rising continuously: persistent issue

Load value exceeds 2x CPU core count

Judgment Logic

First confirm CPU core count:

grep -c processor /proc/cpuinfo
# or
nproc

If output is 4, it's a 4-core server. If Load Average is 8.56, average 8.56 processes wait for CPU or I/O, exceeding 2x core count — severe congestion.

Next Action

Proceed to check CPU usage.

Step 2: Check CPU Usage

Purpose

Confirm whether high Load Average is caused by high CPU usage.

Command

top

Focus on the third line %Cpu(s):

%Cpu(s): 85.2 us, 10.5 sy, 0.0 ni, 2.3 id, 1.5 wa, 0.0 hi, 0.5 si, 0.0 st

Field meanings: us: user-space CPU usage sy: kernel-space CPU usage ni: user-space CPU with nice priority id: idle CPU wa: CPU waiting for I/O hi: hardware interrupt usage si: software interrupt usage st: CPU time stolen by hypervisor (VM)

Abnormal Indicators

us

or sy very high (>80%): CPU-intensive tasks wa very high (>30%): disk I/O bottleneck si very high (>20%): high network traffic or NIC interrupt overload id very low (<10%): CPU near saturation

Judgment Logic

If id near 0, CPU is saturated; need to identify which processes consume CPU. If wa high, many processes wait for I/O; check disk. If si high, network interrupt handling consumes CPU; check network traffic or NIC config.

Next Action

If CPU usage high, jump to Step 7 to identify processes. If wa high, continue Step 4 for disk I/O. If CPU usage not high but Load Average high, likely memory or I/O issue; continue Step 3.

Step 3: Check Memory Usage

Purpose

Confirm if insufficient memory causes frequent swap, slowing the system.

Command

free -h

Expected Output

total        used        free      shared  buff/cache   available
Mem:           7.6G        5.2G        200M        100M        2.2G        2.1G
Swap:          2.0G        1.8G        200M

Abnormal Indicators

Swap usage >50% available memory <500M used close to

total

Judgment Logic

If Swap usage large and available memory low, memory is insufficient. Further confirm frequent swap in/out: vmstat 1 5 Watch si and so columns:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  2 1843200 204800 102400 2252800 5120 3840  1024  2048 8000 12000 65 20 10 5  0
si

: KB/sec swapped in from disk so: KB/sec swapped out to disk

If si and so continuously non-zero and large, system is frequently swapping, severely degrading performance.

Next Action

If memory shortage confirmed: - Identify processes consuming most memory - Consider memory expansion - Optimize application memory usage - Adjust swap strategy Continue Step 7 to identify specific processes.

Step 4: Check Disk I/O

Purpose

Confirm if disk I/O bottleneck causes many processes in D state.

Command

iostat -x 1 5

Expected Output

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda             50.2   120.5   2048.3    4096.7     5.2     10.5   9.38   8.02   15.3    25.6   3.25    40.8     34.0    8.5   95.2

Key metrics: %util: disk busy percentage; near 100% indicates saturation await: average I/O request wait time (ms); >20ms needs attention r_await: read request average wait time w_await: write request average wait time aqu-sz: average queue length; >1 indicates queuing

Abnormal Indicators

%util

near or at 100% await >50ms aqu-sz >5

Judgment Logic

If %util near 100% and await high, disk is bottleneck. Need to identify which processes perform heavy I/O:

iotop -o
-o

shows only processes currently doing I/O.

Expected Output

Total DISK READ: 50.2 M/s | Total DISK WRITE: 80.5 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 1234  be/4  mysql      30.5 M/s    50.2 M/s  0.00 %  85.23 %  mysqld
 5678  be/4  root       15.3 M/s    20.1 M/s  0.00 %  45.67 %  rsync
mysqld

and rsync are performing heavy disk I/O.

Next Action

If disk I/O bottleneck confirmed: - Identify specific processes and I/O types - Check for large file operations, backups, log writes - Optimize application I/O patterns - Consider upgrading to SSD - Check filesystem anomalies Continue Step 7 to identify processes.

Step 5: Check Network Traffic

Purpose

Confirm if high network traffic causes high soft interrupts, consuming CPU.

Command

sar -n DEV 1 5

Expected Output

14:35:01     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
14:35:02        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
14:35:02      eth0  15234.50  18567.20  45678.30  67890.50      0.00      0.00      0.00

Watch: rxpck/s: packets received per second txpck/s: packets transmitted per second rxkB/s: KB received per second txkB/s: KB transmitted per second

Abnormal Indicators

Packet rate >100k/sec

Traffic near bandwidth limit

Sudden traffic spike

Judgment Logic

High network traffic may cause high NIC soft interrupts. Further check soft interrupts: cat /proc/softirqs Focus on NET_RX and NET_TX lines for network receive/transmit soft interrupt counts. Rapid growth indicates high network soft interrupts. Also check top for CPU si percentage:

top
# Observe %Cpu(s) line si value

If si >20%, soft interrupts consume significant CPU.

Next Action

If network traffic confirmed as cause: - Identify processes with heavy network I/O - Check for attack traffic - Check for abnormal connections - Optimize application network patterns - Consider enabling NIC multi-queue, RPS, RFS Continue Step 7 to identify processes.

Step 6: Check Process State Distribution

Purpose

Count processes in each state, focusing on R and D states.

Command

ps -eo stat | sort | uniq -c | sort -rn

Expected Output

45 S
    12 R
     8 D
     5 I
     2 Z
     1 STAT

State meanings: R: running or runnable S: interruptible sleep D: uninterruptible sleep (waiting for I/O) Z: zombie T: stopped or traced I: idle kernel thread

Abnormal Indicators

R

state processes far exceed CPU core count D state processes >5 Z state processes >10

Judgment Logic

Many D state processes indicate many processes waiting for I/O, raising Load Average without necessarily high CPU. Many R state processes indicate busy CPU scheduling; need to identify contending processes. Many Z state processes indicate application logic issues (though they don't consume resources).

Next Action

Continue Step 7 to identify specific processes.

Step 7: Identify Specific Processes

Purpose

Find processes consuming most resources or in abnormal states.

Method 1: top for CPU usage

top
# Press P to sort by CPU usage
# Press M to sort by memory usage

Record PIDs and commands of top CPU-consuming processes.

Method 2: ps for all processes

ps aux --sort=-%cpu | head -20
ps aux --sort=-%mem | head -20

Method 3: View D state processes

ps aux | awk '$8 ~ /D/ {print $0}'

Method 4: View zombie processes

ps aux | awk '$8 ~ /Z/ {print $0}'

Expected Output

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
mysql     1234 85.2 15.3 2048000 1024000 ?      R    10:30  120:45 /usr/sbin/mysqld
www       5678 25.6  8.5 1024000  524288 ?      R    12:15   45:20 /usr/bin/php-fpm

Abnormal Indicators

Process CPU usage >80% sustained

Process memory usage >50%

Multiple processes in D state

Large number of zombie processes

Judgment Logic

Record abnormal process PID, user, command, resource usage; proceed to analyze behavior.

Next Action

Continue Step 8 to analyze process behavior.

Step 8: Analyze Process Behavior

Purpose

Deeply analyze abnormal process behavior to confirm root cause.

Method 1: View open files

lsof -p <PID>

Shows files, sockets, devices opened. Focus on: many open files, file handle leaks, remote filesystem access, abnormal network connections.

Method 2: View system calls

strace -p <PID> -c -f
-c

counts syscall frequency and time; -f follows child processes. Stop with Ctrl+C for summary:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 65.23    2.345678       123456        19          0 read
 20.45    0.734567       12345         60          0 write
  8.12    0.291234        5678         51          0 poll
  3.20    0.115012        2345         49          0 futex
  2.00    0.071890        1234         58          0 epoll_wait

High read / write percentage and latency indicates heavy I/O. High futex / epoll_wait indicates waiting for locks or events.

Method 3: View process stack

pstack <PID>

Shows function call stack, revealing where process is stuck.

Method 4: View process I/O stats

cat /proc/<PID>/io

Example output:

rchar: 123456789
wchar: 987654321
syscr: 12345
syscw: 54321
read_bytes: 102400000
write_bytes: 204800000
cancelled_write_bytes: 0
rchar

: characters read wchar: characters written read_bytes: actual bytes read from disk write_bytes: actual bytes written to disk

Large read_bytes / write_bytes indicates heavy disk I/O.

Method 5: View network connections

lsof -i -a -p <PID>
# or
netstat -antp | grep <PID>
# or
ss -antp | grep <PID>

Method 6: View process threads

ps -eLf | grep <PID>
# or
top -H -p <PID>
-H

shows threads. Reveals thread count, per-thread CPU/memory.

Judgment Logic

Synthesize findings: - Heavy disk I/O: likely log writes, DB queries, file ops - Heavy network I/O: network requests, data transfer - High CPU with few syscalls: compute-intensive tasks - Many network connections: connection pool exhaustion, leaks - Many open files: file handle leaks

Next Action

Apply remediation based on analysis.

Step 9: Apply Remediation Measures

Scenario 1: Process CPU Too High

Temporary Measure

Lower process priority: renice +10 -p <PID> Reduces scheduling priority, giving other processes more CPU time.

Root Measures

Optimize application code logic

Check for infinite loops, excessive computation

Implement rate limiting, degradation, circuit breaking

Consider scaling or load balancing

Risk Warning

renice

only temporary relief, not root fix. Killing process requires confirming business impact and restart plan.

Scenario 2: Memory Shortage Causing Frequent Swap

Temporary Measure

Clear caches:

sync
echo 3 > /proc/sys/vm/drop_caches
sync

ensures dirty pages written to disk; echo 3 clears page cache, dentries, inodes.

Risk Warning

Cache clearing slows subsequent file I/O (re-read from disk). Use cautiously in production, only when memory critically low and cache confirmed excessive.

Root Measures

Identify top memory-consuming processes

Optimize application memory usage

Consider memory expansion

Adjust swappiness:

cat /proc/sys/vm/swappiness
# Default usually 60
# Temporary adjust
echo 10 > /proc/sys/vm/swappiness
# Permanent adjust
echo "vm.swappiness = 10" >> /etc/sysctl.conf
sysctl -p

Lower swappiness makes kernel prefer physical memory over swap. Too low may worsen performance when memory exhausted.

Scenario 3: Disk I/O Bottleneck

Temporary Measure

Pause or throttle large file ops, backups, log archiving. If specific process causes it, lower its I/O priority:

ionice -c3 -p <PID>
-c3

= Idle scheduling class, runs only when no other I/O requests.

Root Measures

Optimize application I/O patterns, reduce random reads/writes

Replace HDD with SSD

Increase disk cache

Check filesystem fragmentation

Check for slow queries causing high DB I/O

Check log rotation config to avoid oversized log files

Scenario 4: High Network Traffic Causing High Soft Interrupts

Temporary Measure

Throttle network traffic or pause non-critical services.

Root Measures

Enable NIC multi-queue:

ethtool -l eth0
# View queue count
ethtool -L eth0 combined 4
# Set to 4 queues

Enable RPS (Receive Packet Steering):

echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus

Enable RFS (Receive Flow Steering):

echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
echo 2048 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt

Optimize application network model, reduce short connections

Consider load balancing to distribute traffic

Scenario 5: D State Process Accumulation

Cause Analysis

Processes in D state wait for I/O completion. Common causes:

Disk failure or slow disk

NFS mount timeout or hang

iSCSI storage anomalies

Driver or kernel bugs

Troubleshooting Method

View D state process stack: cat /proc/<PID>/stack Example output:

[<0>] wait_on_page_bit_common+0x123/0x456
[<0>] __filemap_get_folio+0x234/0x567
[<0>] ext4_write_begin+0x345/0x678
[<0>] generic_perform_write+0x456/0x789
[<0>] ext4_buffered_write_iter+0x567/0x890
[<0>] ext4_file_write_iter+0x678/0x901
[<0>] new_sync_write+0x789/0x012
[<0>] vfs_write+0x890/0x123
[<0>] ksys_write+0x901/0x234
[<0>] __x64_sys_write+0x012/0x345
[<0>] do_syscall_64+0x123/0x456
[<0>] entry_SYSCALL_64_after_hwframe+0x234/0x567

Shows process stuck in file write operation.

Temporary Measure

If NFS mount issue, try force unmount:

umount -f /mnt/nfs
# or
umount -l /mnt/nfs
-f

force unmount; -l lazy unmount.

Risk Warning

Force unmount may cause data loss. Use cautiously. If D state processes cannot be cleared, may require system reboot.

Root Measures

Check disk health: smartctl -a /dev/sda Check NFS config and network connectivity

Check storage device health

Consider replacing faulty hardware

Scenario 6: Zombie Process Accumulation

Cause Analysis

Zombie processes are exited children whose parent hasn't called wait() or waitpid() to reap them. They don't consume CPU/memory but occupy PIDs; excessive zombies may prevent new process creation.

Troubleshooting Method

Find zombie parent processes:

ps -ef | grep defunct
# or
ps -eo pid,ppid,stat,comm | awk '$3 ~ /Z/ {print $0}'

Record parent PID (PPID column).

Remediation

Send SIGCHLD to parent to trigger reaping: kill -s SIGCHLD <parent_PID> If parent bug prevents reaping, restart parent:

kill <parent_PID>
# or force kill
kill -9 <parent_PID>

Risk Warning

Restart parent after confirming business impact.

Root Measure

Fix application code to properly handle child process exit.

Step 10: Verify Effectiveness

Purpose

Confirm remediation works, load returns to normal.

Commands

Re-check Load Average: uptime Re-check CPU, memory, disk, network:

top
free -h
iostat -x 1 5
sar -n DEV 1 5

Expected Results

Load Average drops to normal range

CPU idle percentage rises

Disk %util decreases

Memory swap usage drops

D state process count decreases

Abnormal Indicators

If load remains high, issue unresolved or other root causes exist; continue troubleshooting.

Next Action

If resolved, proceed to Step 11 post-mortem. If not, return to Step 2.

Step 11: Post-Mortem Summary

Purpose

Document troubleshooting process, root cause, remediation, verification for future reference and improvement.

Key Points

Problem trigger time and symptoms

Monitoring alert content

Troubleshooting path and key commands

Identified root cause

Remediation actions taken

Verification results

Whether monitoring alerts need optimization

Whether application architecture needs optimization

Whether scaling or hardware upgrade needed

Output

Produce incident report or knowledge base document including:

Problem description

Impact scope

Root cause analysis

Remediation measures

Preventive measures

Lessons learned

6. Common Commands

Commands for troubleshooting high Load Average, categorized by scenario.

6.1 View Load and Uptime

uptime

6.2 View CPU Info

# CPU core count
grep -c processor /proc/cpuinfo
nproc
# CPU details
lscpu
# CPU usage
top
htop
# CPU history
sar -u 1 5

6.3 View Memory Info

# Memory usage
free -h
# Detailed memory stats
cat /proc/meminfo
# Swap usage
swapon -s
# Swap statistics
vmstat 1 5

6.4 View Disk I/O Info

# Disk I/O stats
iostat -x 1 5
# Per-process disk I/O
iotop -o
# Disk usage
df -h
# Inode usage
df -i
# Disk health
smartctl -a /dev/sda

6.5 View Network Info

# Network traffic
sar -n DEV 1 5
ifstat
nload
# Network connections
netstat -antp
ss -antp
# NIC queues
ethtool -l eth0
# Soft interrupt stats
cat /proc/softirqs

6.6 View Process Info

# All processes
ps aux
ps -ef
# Sort by CPU
ps aux --sort=-%cpu | head -20
# Sort by memory
ps aux --sort=-%mem | head -20
# Process state distribution
ps -eo stat | sort | uniq -c | sort -rn
# D state processes
ps aux | awk '$8 ~ /D/ {print $0}'
# Zombie processes
ps aux | awk '$8 ~ /Z/ {print $0}'
# Process tree
pstree -p
# Process details
top -p <PID>
htop -p <PID>

6.7 Analyze Process Behavior

# Open files
lsof -p <PID>
# Network connections
lsof -i -a -p <PID>
# System calls
strace -p <PID> -c -f
# Process stack
pstack <PID>
cat /proc/<PID>/stack
# I/O stats
cat /proc/<PID>/io
# Threads
ps -eLf | grep <PID>
top -H -p <PID>
# Process status
cat /proc/<PID>/status
# Command line
cat /proc/<PID>/cmdline
# Environment variables
cat /proc/<PID>/environ

6.8 Adjust Process Priority

# Lower CPU priority
renice +10 -p <PID>
# Lower I/O priority
ionice -c3 -p <PID>

6.9 Memory Management

# Clear caches (caution)
sync
echo 3 > /proc/sys/vm/drop_caches
# View and adjust swappiness
cat /proc/sys/vm/swappiness
echo 10 > /proc/sys/vm/swappiness
# View huge pages
cat /proc/meminfo | grep -i huge

6.10 System Info

# Kernel version
uname -r
# OS version
cat /etc/os-release
# System logs
journalctl -xe
dmesg | tail -50
# Load history
sar -q 1 5

7. Configuration Examples

7.1 Optimize Swap Usage Policy

Edit /etc/sysctl.conf:

# Reduce swap tendency
vm.swappiness = 10
# Increase dirty page flush frequency
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

Apply:

sysctl -p

7.2 Enable NIC Multi-Queue

View supported queues: ethtool -l eth0 Example output:

Channel parameters for eth0:
Pre-set maximums:
RX:          4
TX:          4
Other:       0
Combined:    4
Current hardware settings:
RX:          1
TX:          1
Other:       0
Combined:    1

Set to max queues:

ethtool -L eth0 combined 4

7.3 Enable RPS and RFS

RPS config (4-core example):

# Set RPS CPU mask (0xf = CPUs 0-3)
echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus

RFS config:

# Global flow table size
echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
# Per-queue flow table size
echo 2048 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt

7.4 Adjust File Descriptor Limits

Edit /etc/security/limits.conf:

* soft nofile 65536
* hard nofile 65536
* soft nproc 32768
* hard nproc 32768

Edit /etc/sysctl.conf:

# System-wide file descriptor limit
fs.file-max = 6553560

Apply: sysctl -p. Re-login for limits.conf.

7.5 Adjust Kernel Parameters

Edit /etc/sysctl.conf:

# Increase max connections
net.core.somaxconn = 32768
# Increase TCP connection queue
net.ipv4.tcp_max_syn_backlog = 8192
# Enable TCP timestamps
net.ipv4.tcp_timestamps = 1
# Enable TCP window scaling
net.ipv4.tcp_window_scaling = 1
# Reduce TIME_WAIT connections
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
# Increase local port range
net.ipv4.ip_local_port_range = 10000 65000

Apply:

sysctl -p

8. Log and Metric Observation Methods

8.1 System Logs

# Last 50 lines of system log
journalctl -xe | tail -50
# Kernel log
dmesg | tail -50
# Follow system log
journalctl -f

Watch keywords: Out of memory: memory exhaustion I/O error: disk I/O error segfault: process segmentation fault hung task: process hang soft lockup: soft lockup hard lockup: hard lockup

8.2 Application Logs

Common paths:

Nginx: /var/log/nginx/error.log Apache: /var/log/httpd/error_log MySQL: /var/log/mysql/error.log Redis: /var/log/redis/redis-server.log PHP-FPM:

/var/log/php-fpm/error.log

8.3 Monitoring Metrics

If using Prometheus + Grafana, watch: node_load1, node_load5, node_load15: Load Average node_cpu_seconds_total: CPU time node_memory_MemAvailable_bytes: available memory node_memory_SwapCached_bytes: swap usage node_disk_io_time_seconds_total: disk I/O time node_disk_read_bytes_total: disk read bytes node_disk_written_bytes_total: disk write bytes node_network_receive_bytes_total: network receive bytes node_network_transmit_bytes_total: network transmit bytes

Metric names depend on exporter version.

9. Troubleshooting Paths

Complete paths for typical scenarios.

Scenario 1: High Load Average with High CPU Usage

Step 1: uptime confirm high load
Step 2: top confirm high CPU usage
Step 3: ps aux --sort=-%cpu find top CPU processes
Step 4: lsof -p <PID> view open files and connections
Step 5: strace -p <PID> -c view syscalls
Step 6: Analyze process behavior, confirm normalcy
Step 7: If compute-intensive, optimize code or scale
Step 8: If abnormal process, consider restart or kill
Step 9: Verify load recovery

Scenario 2: High Load Average but CPU Idle

Step 1: uptime confirm high load
Step 2: top confirm high CPU idle but wa high
Step 3: iostat -x 1 5 confirm disk I/O bottleneck
Step 4: iotop -o find top I/O processes
Step 5: lsof -p <PID> view accessed files
Step 6: Check for large file ops, backups
Step 7: If excessive log writes, adjust log level or rotation
Step 8: If DB slow queries, optimize SQL or add indexes
Step 9: Verify load recovery

Scenario 3: Multiple D State Processes

Step 1: ps aux | awk '$8 ~ /D/ {print $0}' find D state processes
Step 2: cat /proc/<PID>/stack view process stack
Step 3: Confirm which syscall process stuck on
Step 4: If disk I/O, check disk health
Step 5: If NFS mount, check network and NFS service
Step 6: Try force unmount or restart related services
Step 7: If unresolved, may need system reboot

Scenario 4: High Soft Interrupts

Step 1: top confirm high si percentage
Step 2: cat /proc/softirqs confirm network soft interrupts
Step 3: sar -n DEV 1 5 confirm high network traffic
Step 4: netstat -antp or ss -antp view connections
Step 5: Identify top network-consuming processes
Step 6: ethtool -l eth0 check NIC queue config
Step 7: Enable NIC multi-queue, RPS, RFS
Step 8: Verify soft interrupt reduction

Scenario 5: Memory Shortage with Frequent Swap

Step 1: free -h confirm low memory, high swap usage
Step 2: vmstat 1 5 confirm frequent si and so
Step 3: ps aux --sort=-%mem find top memory processes
Step 4: Analyze process memory usage reasonableness
Step 5: Optimize app memory or expand memory
Step 6: Temporarily clear caches (caution)
Step 7: Adjust swappiness parameter
Step 8: Verify swap usage reduction

10. Risk Warnings

10.1 Cache Clearing Risks

echo 3 > /proc/sys/vm/drop_caches

clears page cache, dentries, inodes, may cause:

Subsequent file I/O slows (re-read from disk)

Ongoing I/O operations affected

Database performance temporarily degrades

Recommendations: avoid in production; only when memory critically low and cache confirmed excessive; execute during low traffic.

10.2 Force Kill Process Risks

kill -9 <PID>

may cause:

Data loss or corruption

Lock files not cleaned

Child processes become orphans

Service fails to restart cleanly

Recommendations: prefer kill <PID> for graceful exit; confirm business impact and restart plan; check backups.

10.3 Force Unmount Filesystem Risks

umount -f

or umount -l may cause:

Processes accessing filesystem crash

Data loss

Filesystem corruption

Recommendations: confirm no processes using filesystem; use lsof /mnt/path or fuser -m /mnt/path; stop related services first.

10.4 Kernel Parameter Adjustment Risks

Wrong /etc/sysctl.conf parameters may cause:

System instability

Network anomalies

Performance degradation

Recommendations: test in staging first; backup original config; adjust one parameter at a time; document reason and expected effect.

10.5 Service Restart Risks

Restarting services may cause:

Business interruption

User request failures

Data loss

Recommendations: assess business impact first; choose low-traffic window; notify stakeholders; prepare rollback plan; monitor service post-restart.

11. Verification Methods

11.1 Verify Load Recovery

uptime
# Observe Load Average drop to normal range

Observe continuously for minutes to confirm sustained drop or stability.

11.2 Verify CPU Usage Normal

top
# Observe %Cpu(s) id value rise
# Observe wa, si values drop

11.3 Verify Memory Improvement

free -h
# Observe available memory increase
# Observe swap usage decrease

11.4 Verify Disk I/O Relief

iostat -x 1 5
# Observe %util drop
# Observe await drop

11.5 Verify Network Traffic Normal

sar -n DEV 1 5
# Observe rxpck/s, txpck/s return to normal levels

11.6 Verify Soft Interrupt Reduction

top
# Observe %Cpu(s) si value drop

11.7 Verify Process States Normal

ps -eo stat | sort | uniq -c | sort -rn
# Observe D state process reduction
# Observe R state processes in reasonable range

11.8 Verify Business Recovery

Check application logs for errors

Check user feedback normalization

Check monitoring dashboards

Perform functional business validation

12. Rollback Plans

12.1 Kernel Parameter Rollback

If /etc/sysctl.conf changes cause issues:

# Restore backup config
cp /etc/sysctl.conf.bak /etc/sysctl.conf
# Apply
sysctl -p
# Or revert single parameter
# Restore swappiness default
echo 60 > /proc/sys/vm/swappiness

12.2 NIC Config Rollback

If NIC queue or RPS changes cause network issues:

# Restore default queue count
ethtool -L eth0 combined 1
# Clear RPS config
echo 0 > /sys/class/net/eth0/queues/rx-0/rps_cpus

12.3 Service Restart Rollback

If service restart worsens problem:

Roll back to previous stable version via version management

Restore old config from backup

Restore data from backup

Consider snapshot restore if available

12.4 Process Kill Rollback

If killing process impacts business:

# Restart service
systemctl start <service_name>
# Or manual start
/path/to/binary &

Confirm after restart: process running, logs error-free, business functional.

13. Production Environment Considerations

13.1 Pre-Operation Checks

Confirm not business peak hours

Confirm no critical tasks running

Confirm backups exist

Confirm rollback plan feasible

Confirm relevant personnel notified

13.2 During Operation Monitoring

Continuously watch monitoring dashboards

Continuously watch application logs

Continuously watch system logs

Be ready to rollback anytime

13.3 Post-Operation Verification

Verify load recovery

Verify business normal

Verify monitoring normal

Document operation process and results

13.4 Permission Control

Production operations require approval

Critical operations need two-person confirmation

Record operation logs

Avoid direct root usage

13.5 Change Management

Follow change process

Record change content

Record change reason

Record change impact

Record rollback plan

13.6 Emergency Preparedness

Maintain emergency contact list

Prepare emergency handling procedures

Prepare common command checklist

Regularly drill emergency processes

14. Summary

Troubleshooting high Linux system load is a fundamental skill for operations engineers, requiring mastery of:

14.1 Understand Core Concepts

Load Average reflects runnable and uninterruptible sleep process counts

Must combine with CPU core count to judge normality

High Load Average ≠ high CPU; could be I/O bottleneck

14.2 Establish Troubleshooting Path

Overall first, then local

Observe first, then act

Eliminate obvious issues first

Non-intrusive first, then intrusive

Record the scene

14.3 Master Tools

uptime

, top, htop: load and CPU free, vmstat: memory and swap iostat, iotop: disk I/O sar, netstat, ss: network ps, lsof, strace, pstack: process analysis

14.4 Master Analysis Methods

Categorize by CPU, memory, disk, network

Focus on process state distribution, especially R and D

Deep-dive into abnormal process syscalls, open files, network connections

Correlate multiple metrics to determine root cause

14.5 Apply Effective Measures

Temporary: lower priority, clear caches, throttle

Root: optimize code, adjust config, scale resources

High-risk ops require impact assessment and rollback prep

Production ops must follow change process

14.6 Close the Loop

Must verify effectiveness after fix

Prepare rollback plan

Document troubleshooting process and results

Produce incident reports and knowledge base

Optimize monitoring alerts and prevention mechanisms

Only through extensive practical accumulation can one quickly locate issues, take correct actions, and restore business during production incidents. This article provides a systematic troubleshooting methodology and toolset, to be flexibly applied per specific business scenarios and system characteristics.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

NetworkPerformance TuningLinuxTroubleshootingCPUMemorySystem AdministrationDisk I/OLoad AverageProcess Analysis
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.