Master Linux Load Average Troubleshooting: From Concepts to Real-World Fixes
This comprehensive guide covers Linux Load Average fundamentals, diagnostic commands (top, iostat, vmstat, strace), step-by-step troubleshooting for CPU, memory, disk I/O, network, and process issues, plus remediation tactics, verification methods, and production safety practices.
1. Problem Background
One of the most common operational scenarios in production environments is a sudden spike in server load, causing slow or unavailable business responses. Monitoring platforms frequently alert on high Load Average, and SSH logins become sluggish. Quickly identifying the root cause and taking effective measures to restore service tests an engineer's depth of understanding of the Linux resource model and practical experience.
High Load Average does not necessarily mean high CPU usage or insufficient memory. Real causes may include disk I/O bottlenecks, network traffic surges, zombie process accumulation, excessive soft interrupts, kernel task hangs, and more. Therefore, troubleshooting load issues requires a complete observation chain and judgment logic to avoid blindly restarting services or tuning parameters by guesswork.
This article systematically organizes Linux load core concepts, troubleshooting paths, tool usage, command interpretation, common root causes, and remediation measures from a practical perspective, aiming to form a complete troubleshooting loop.
2. Applicable Scenarios
The content applies to the following typical scenarios:
Monitoring alerts show server Load Average persistently above normal levels
Users report slow or timed-out business responses
SSH login to the server is noticeably laggy
Application logs show large numbers of slow requests or queue backlogs
Database queries slow down, connection counts surge
Container node resource usage is abnormal
Cron jobs or batch processing cause system resource contention
Performance degrades after system upgrades, scaling, or migrations
Whether physical machines, virtual machines, or cloud instances, as long as they run the Linux kernel, the troubleshooting approaches and commands in this article apply.
3. Core Knowledge Points
3.1 What is Load Average
Load Average is a Linux metric measuring system busyness, representing the average number of processes in runnable (R state) and uninterruptible sleep (D state) over a specific time interval.
Typically three values are shown, representing the 1-minute, 5-minute, and 15-minute average loads.
uptime
# Output example:
# 14:23:45 up 10 days, 3:15, 2 users, load average: 2.35, 1.80, 1.50The values 2.35, 1.80, 1.50 are the Load Average.
R State Processes
Processes in Running or Runnable state, either using CPU or waiting for CPU scheduling.
D State Processes
Processes in uninterruptible sleep (Uninterruptible Sleep), usually waiting for I/O operations to complete, such as disk reads/writes or NFS mount hangs. These processes cannot be interrupted by signals or killed; they can only wait for I/O completion or a system reboot.
3.2 How High is High for Load Average
There is no absolute standard; judgment must combine CPU core count.
General rules:
Load Average close to or less than CPU core count: system normal
Load Average 1.5x CPU core count or more: needs attention
Load Average 2x CPU core count or more: severe congestion
For example, a 4-core server:
Load Average below 3: normal
Load Average between 4-6: needs investigation
Load Average above 8: severe congestion
These are empirical values; business characteristics and historical baselines must also be considered.
3.3 Possible Causes of High Load Average
Excessive CPU-intensive tasks
Disk I/O bottlenecks
High network I/O
Insufficient memory triggering swap
Excessive soft interrupts
Zombie process accumulation
Kernel task hangs
NFS, CIFS, or other remote filesystem mount anomalies
Driver or hardware failures
3.4 Troubleshooting Principles
Overall first, then local: check system-wide metrics first, then specific processes
Observe first, then act: gather sufficient information before deciding
Eliminate obvious issues first: prioritize checking CPU, memory, disk, network
Non-intrusive first, then intrusive: prefer read-only commands, avoid immediate service restarts
Record the scene: save key command outputs for post-mortem analysis
4. Overall Troubleshooting Approach
Recommended flow for troubleshooting high Load Average:
Step 1: Confirm load is truly abnormal
Step 2: Check CPU usage
Step 3: Check memory usage
Step 4: Check disk I/O
Step 5: Check network traffic
Step 6: Check process state distribution
Step 7: Identify specific processes
Step 8: Analyze process behavior
Step 9: Apply remediation measures
Step 10: Verify effectiveness
Step 11: Post-mortem summary5. Practical Steps
Step 1: Confirm Load is Abnormal
Purpose
Avoid misjudgment; first confirm Load Average indeed exceeds normal range.
Command
uptimeExpected Output
14:30:12 up 10 days, 3:21, 2 users, load average: 8.56, 7.23, 6.10Abnormal Indicators
1-minute load far higher than 5-minute and 15-minute: burst issue
All three load values high and rising continuously: persistent issue
Load value exceeds 2x CPU core count
Judgment Logic
First confirm CPU core count:
grep -c processor /proc/cpuinfo
# or
nprocIf output is 4, it's a 4-core server. If Load Average is 8.56, average 8.56 processes wait for CPU or I/O, exceeding 2x core count — severe congestion.
Next Action
Proceed to check CPU usage.
Step 2: Check CPU Usage
Purpose
Confirm whether high Load Average is caused by high CPU usage.
Command
topFocus on the third line %Cpu(s):
%Cpu(s): 85.2 us, 10.5 sy, 0.0 ni, 2.3 id, 1.5 wa, 0.0 hi, 0.5 si, 0.0 stField meanings: us: user-space CPU usage sy: kernel-space CPU usage ni: user-space CPU with nice priority id: idle CPU wa: CPU waiting for I/O hi: hardware interrupt usage si: software interrupt usage st: CPU time stolen by hypervisor (VM)
Abnormal Indicators
usor sy very high (>80%): CPU-intensive tasks wa very high (>30%): disk I/O bottleneck si very high (>20%): high network traffic or NIC interrupt overload id very low (<10%): CPU near saturation
Judgment Logic
If id near 0, CPU is saturated; need to identify which processes consume CPU.
If wa high, many processes wait for I/O; check disk.
If si high, network interrupt handling consumes CPU; check network traffic or NIC config.
Next Action
If CPU usage high, jump to Step 7 to identify processes.
If wa high, continue Step 4 for disk I/O.
If CPU usage not high but Load Average high, likely memory or I/O issue; continue Step 3.
Step 3: Check Memory Usage
Purpose
Confirm if insufficient memory causes frequent swap, slowing the system.
Command
free -hExpected Output
total used free shared buff/cache available
Mem: 7.6G 5.2G 200M 100M 2.2G 2.1G
Swap: 2.0G 1.8G 200MAbnormal Indicators
Swap usage >50% available memory <500M used close to
totalJudgment Logic
If Swap usage large and available memory low, memory is insufficient.
Further confirm frequent swap in/out: vmstat 1 5 Watch si and so columns:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 2 1843200 204800 102400 2252800 5120 3840 1024 2048 8000 12000 65 20 10 5 0si: KB/sec swapped in from disk so: KB/sec swapped out to disk
If si and so continuously non-zero and large, system is frequently swapping, severely degrading performance.
Next Action
If memory shortage confirmed: - Identify processes consuming most memory - Consider memory expansion - Optimize application memory usage - Adjust swap strategy Continue Step 7 to identify specific processes.
Step 4: Check Disk I/O
Purpose
Confirm if disk I/O bottleneck causes many processes in D state.
Command
iostat -x 1 5Expected Output
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
sda 50.2 120.5 2048.3 4096.7 5.2 10.5 9.38 8.02 15.3 25.6 3.25 40.8 34.0 8.5 95.2Key metrics: %util: disk busy percentage; near 100% indicates saturation await: average I/O request wait time (ms); >20ms needs attention r_await: read request average wait time w_await: write request average wait time aqu-sz: average queue length; >1 indicates queuing
Abnormal Indicators
%utilnear or at 100% await >50ms aqu-sz >5
Judgment Logic
If %util near 100% and await high, disk is bottleneck.
Need to identify which processes perform heavy I/O:
iotop -o-oshows only processes currently doing I/O.
Expected Output
Total DISK READ: 50.2 M/s | Total DISK WRITE: 80.5 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
1234 be/4 mysql 30.5 M/s 50.2 M/s 0.00 % 85.23 % mysqld
5678 be/4 root 15.3 M/s 20.1 M/s 0.00 % 45.67 % rsyncmysqldand rsync are performing heavy disk I/O.
Next Action
If disk I/O bottleneck confirmed: - Identify specific processes and I/O types - Check for large file operations, backups, log writes - Optimize application I/O patterns - Consider upgrading to SSD - Check filesystem anomalies Continue Step 7 to identify processes.
Step 5: Check Network Traffic
Purpose
Confirm if high network traffic causes high soft interrupts, consuming CPU.
Command
sar -n DEV 1 5Expected Output
14:35:01 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
14:35:02 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
14:35:02 eth0 15234.50 18567.20 45678.30 67890.50 0.00 0.00 0.00Watch: rxpck/s: packets received per second txpck/s: packets transmitted per second rxkB/s: KB received per second txkB/s: KB transmitted per second
Abnormal Indicators
Packet rate >100k/sec
Traffic near bandwidth limit
Sudden traffic spike
Judgment Logic
High network traffic may cause high NIC soft interrupts.
Further check soft interrupts: cat /proc/softirqs Focus on NET_RX and NET_TX lines for network receive/transmit soft interrupt counts. Rapid growth indicates high network soft interrupts.
Also check top for CPU si percentage:
top
# Observe %Cpu(s) line si valueIf si >20%, soft interrupts consume significant CPU.
Next Action
If network traffic confirmed as cause: - Identify processes with heavy network I/O - Check for attack traffic - Check for abnormal connections - Optimize application network patterns - Consider enabling NIC multi-queue, RPS, RFS Continue Step 7 to identify processes.
Step 6: Check Process State Distribution
Purpose
Count processes in each state, focusing on R and D states.
Command
ps -eo stat | sort | uniq -c | sort -rnExpected Output
45 S
12 R
8 D
5 I
2 Z
1 STATState meanings: R: running or runnable S: interruptible sleep D: uninterruptible sleep (waiting for I/O) Z: zombie T: stopped or traced I: idle kernel thread
Abnormal Indicators
Rstate processes far exceed CPU core count D state processes >5 Z state processes >10
Judgment Logic
Many D state processes indicate many processes waiting for I/O, raising Load Average without necessarily high CPU.
Many R state processes indicate busy CPU scheduling; need to identify contending processes.
Many Z state processes indicate application logic issues (though they don't consume resources).
Next Action
Continue Step 7 to identify specific processes.
Step 7: Identify Specific Processes
Purpose
Find processes consuming most resources or in abnormal states.
Method 1: top for CPU usage
top
# Press P to sort by CPU usage
# Press M to sort by memory usageRecord PIDs and commands of top CPU-consuming processes.
Method 2: ps for all processes
ps aux --sort=-%cpu | head -20
ps aux --sort=-%mem | head -20Method 3: View D state processes
ps aux | awk '$8 ~ /D/ {print $0}'Method 4: View zombie processes
ps aux | awk '$8 ~ /Z/ {print $0}'Expected Output
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mysql 1234 85.2 15.3 2048000 1024000 ? R 10:30 120:45 /usr/sbin/mysqld
www 5678 25.6 8.5 1024000 524288 ? R 12:15 45:20 /usr/bin/php-fpmAbnormal Indicators
Process CPU usage >80% sustained
Process memory usage >50%
Multiple processes in D state
Large number of zombie processes
Judgment Logic
Record abnormal process PID, user, command, resource usage; proceed to analyze behavior.
Next Action
Continue Step 8 to analyze process behavior.
Step 8: Analyze Process Behavior
Purpose
Deeply analyze abnormal process behavior to confirm root cause.
Method 1: View open files
lsof -p <PID>Shows files, sockets, devices opened. Focus on: many open files, file handle leaks, remote filesystem access, abnormal network connections.
Method 2: View system calls
strace -p <PID> -c -f-ccounts syscall frequency and time; -f follows child processes. Stop with Ctrl+C for summary:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
65.23 2.345678 123456 19 0 read
20.45 0.734567 12345 60 0 write
8.12 0.291234 5678 51 0 poll
3.20 0.115012 2345 49 0 futex
2.00 0.071890 1234 58 0 epoll_waitHigh read / write percentage and latency indicates heavy I/O.
High futex / epoll_wait indicates waiting for locks or events.
Method 3: View process stack
pstack <PID>Shows function call stack, revealing where process is stuck.
Method 4: View process I/O stats
cat /proc/<PID>/ioExample output:
rchar: 123456789
wchar: 987654321
syscr: 12345
syscw: 54321
read_bytes: 102400000
write_bytes: 204800000
cancelled_write_bytes: 0rchar: characters read wchar: characters written read_bytes: actual bytes read from disk write_bytes: actual bytes written to disk
Large read_bytes / write_bytes indicates heavy disk I/O.
Method 5: View network connections
lsof -i -a -p <PID>
# or
netstat -antp | grep <PID>
# or
ss -antp | grep <PID>Method 6: View process threads
ps -eLf | grep <PID>
# or
top -H -p <PID>-Hshows threads. Reveals thread count, per-thread CPU/memory.
Judgment Logic
Synthesize findings: - Heavy disk I/O: likely log writes, DB queries, file ops - Heavy network I/O: network requests, data transfer - High CPU with few syscalls: compute-intensive tasks - Many network connections: connection pool exhaustion, leaks - Many open files: file handle leaks
Next Action
Apply remediation based on analysis.
Step 9: Apply Remediation Measures
Scenario 1: Process CPU Too High
Temporary Measure
Lower process priority: renice +10 -p <PID> Reduces scheduling priority, giving other processes more CPU time.
Root Measures
Optimize application code logic
Check for infinite loops, excessive computation
Implement rate limiting, degradation, circuit breaking
Consider scaling or load balancing
Risk Warning
reniceonly temporary relief, not root fix. Killing process requires confirming business impact and restart plan.
Scenario 2: Memory Shortage Causing Frequent Swap
Temporary Measure
Clear caches:
sync
echo 3 > /proc/sys/vm/drop_cachessyncensures dirty pages written to disk; echo 3 clears page cache, dentries, inodes.
Risk Warning
Cache clearing slows subsequent file I/O (re-read from disk). Use cautiously in production, only when memory critically low and cache confirmed excessive.
Root Measures
Identify top memory-consuming processes
Optimize application memory usage
Consider memory expansion
Adjust swappiness:
cat /proc/sys/vm/swappiness
# Default usually 60
# Temporary adjust
echo 10 > /proc/sys/vm/swappiness
# Permanent adjust
echo "vm.swappiness = 10" >> /etc/sysctl.conf
sysctl -pLower swappiness makes kernel prefer physical memory over swap. Too low may worsen performance when memory exhausted.
Scenario 3: Disk I/O Bottleneck
Temporary Measure
Pause or throttle large file ops, backups, log archiving. If specific process causes it, lower its I/O priority:
ionice -c3 -p <PID>-c3= Idle scheduling class, runs only when no other I/O requests.
Root Measures
Optimize application I/O patterns, reduce random reads/writes
Replace HDD with SSD
Increase disk cache
Check filesystem fragmentation
Check for slow queries causing high DB I/O
Check log rotation config to avoid oversized log files
Scenario 4: High Network Traffic Causing High Soft Interrupts
Temporary Measure
Throttle network traffic or pause non-critical services.
Root Measures
Enable NIC multi-queue:
ethtool -l eth0
# View queue count
ethtool -L eth0 combined 4
# Set to 4 queuesEnable RPS (Receive Packet Steering):
echo f > /sys/class/net/eth0/queues/rx-0/rps_cpusEnable RFS (Receive Flow Steering):
echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
echo 2048 > /sys/class/net/eth0/queues/rx-0/rps_flow_cntOptimize application network model, reduce short connections
Consider load balancing to distribute traffic
Scenario 5: D State Process Accumulation
Cause Analysis
Processes in D state wait for I/O completion. Common causes:
Disk failure or slow disk
NFS mount timeout or hang
iSCSI storage anomalies
Driver or kernel bugs
Troubleshooting Method
View D state process stack: cat /proc/<PID>/stack Example output:
[<0>] wait_on_page_bit_common+0x123/0x456
[<0>] __filemap_get_folio+0x234/0x567
[<0>] ext4_write_begin+0x345/0x678
[<0>] generic_perform_write+0x456/0x789
[<0>] ext4_buffered_write_iter+0x567/0x890
[<0>] ext4_file_write_iter+0x678/0x901
[<0>] new_sync_write+0x789/0x012
[<0>] vfs_write+0x890/0x123
[<0>] ksys_write+0x901/0x234
[<0>] __x64_sys_write+0x012/0x345
[<0>] do_syscall_64+0x123/0x456
[<0>] entry_SYSCALL_64_after_hwframe+0x234/0x567Shows process stuck in file write operation.
Temporary Measure
If NFS mount issue, try force unmount:
umount -f /mnt/nfs
# or
umount -l /mnt/nfs-fforce unmount; -l lazy unmount.
Risk Warning
Force unmount may cause data loss. Use cautiously. If D state processes cannot be cleared, may require system reboot.
Root Measures
Check disk health: smartctl -a /dev/sda Check NFS config and network connectivity
Check storage device health
Consider replacing faulty hardware
Scenario 6: Zombie Process Accumulation
Cause Analysis
Zombie processes are exited children whose parent hasn't called wait() or waitpid() to reap them. They don't consume CPU/memory but occupy PIDs; excessive zombies may prevent new process creation.
Troubleshooting Method
Find zombie parent processes:
ps -ef | grep defunct
# or
ps -eo pid,ppid,stat,comm | awk '$3 ~ /Z/ {print $0}'Record parent PID (PPID column).
Remediation
Send SIGCHLD to parent to trigger reaping: kill -s SIGCHLD <parent_PID> If parent bug prevents reaping, restart parent:
kill <parent_PID>
# or force kill
kill -9 <parent_PID>Risk Warning
Restart parent after confirming business impact.
Root Measure
Fix application code to properly handle child process exit.
Step 10: Verify Effectiveness
Purpose
Confirm remediation works, load returns to normal.
Commands
Re-check Load Average: uptime Re-check CPU, memory, disk, network:
top
free -h
iostat -x 1 5
sar -n DEV 1 5Expected Results
Load Average drops to normal range
CPU idle percentage rises
Disk %util decreases
Memory swap usage drops
D state process count decreases
Abnormal Indicators
If load remains high, issue unresolved or other root causes exist; continue troubleshooting.
Next Action
If resolved, proceed to Step 11 post-mortem. If not, return to Step 2.
Step 11: Post-Mortem Summary
Purpose
Document troubleshooting process, root cause, remediation, verification for future reference and improvement.
Key Points
Problem trigger time and symptoms
Monitoring alert content
Troubleshooting path and key commands
Identified root cause
Remediation actions taken
Verification results
Whether monitoring alerts need optimization
Whether application architecture needs optimization
Whether scaling or hardware upgrade needed
Output
Produce incident report or knowledge base document including:
Problem description
Impact scope
Root cause analysis
Remediation measures
Preventive measures
Lessons learned
6. Common Commands
Commands for troubleshooting high Load Average, categorized by scenario.
6.1 View Load and Uptime
uptime6.2 View CPU Info
# CPU core count
grep -c processor /proc/cpuinfo
nproc
# CPU details
lscpu
# CPU usage
top
htop
# CPU history
sar -u 1 56.3 View Memory Info
# Memory usage
free -h
# Detailed memory stats
cat /proc/meminfo
# Swap usage
swapon -s
# Swap statistics
vmstat 1 56.4 View Disk I/O Info
# Disk I/O stats
iostat -x 1 5
# Per-process disk I/O
iotop -o
# Disk usage
df -h
# Inode usage
df -i
# Disk health
smartctl -a /dev/sda6.5 View Network Info
# Network traffic
sar -n DEV 1 5
ifstat
nload
# Network connections
netstat -antp
ss -antp
# NIC queues
ethtool -l eth0
# Soft interrupt stats
cat /proc/softirqs6.6 View Process Info
# All processes
ps aux
ps -ef
# Sort by CPU
ps aux --sort=-%cpu | head -20
# Sort by memory
ps aux --sort=-%mem | head -20
# Process state distribution
ps -eo stat | sort | uniq -c | sort -rn
# D state processes
ps aux | awk '$8 ~ /D/ {print $0}'
# Zombie processes
ps aux | awk '$8 ~ /Z/ {print $0}'
# Process tree
pstree -p
# Process details
top -p <PID>
htop -p <PID>6.7 Analyze Process Behavior
# Open files
lsof -p <PID>
# Network connections
lsof -i -a -p <PID>
# System calls
strace -p <PID> -c -f
# Process stack
pstack <PID>
cat /proc/<PID>/stack
# I/O stats
cat /proc/<PID>/io
# Threads
ps -eLf | grep <PID>
top -H -p <PID>
# Process status
cat /proc/<PID>/status
# Command line
cat /proc/<PID>/cmdline
# Environment variables
cat /proc/<PID>/environ6.8 Adjust Process Priority
# Lower CPU priority
renice +10 -p <PID>
# Lower I/O priority
ionice -c3 -p <PID>6.9 Memory Management
# Clear caches (caution)
sync
echo 3 > /proc/sys/vm/drop_caches
# View and adjust swappiness
cat /proc/sys/vm/swappiness
echo 10 > /proc/sys/vm/swappiness
# View huge pages
cat /proc/meminfo | grep -i huge6.10 System Info
# Kernel version
uname -r
# OS version
cat /etc/os-release
# System logs
journalctl -xe
dmesg | tail -50
# Load history
sar -q 1 57. Configuration Examples
7.1 Optimize Swap Usage Policy
Edit /etc/sysctl.conf:
# Reduce swap tendency
vm.swappiness = 10
# Increase dirty page flush frequency
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10Apply:
sysctl -p7.2 Enable NIC Multi-Queue
View supported queues: ethtool -l eth0 Example output:
Channel parameters for eth0:
Pre-set maximums:
RX: 4
TX: 4
Other: 0
Combined: 4
Current hardware settings:
RX: 1
TX: 1
Other: 0
Combined: 1Set to max queues:
ethtool -L eth0 combined 47.3 Enable RPS and RFS
RPS config (4-core example):
# Set RPS CPU mask (0xf = CPUs 0-3)
echo f > /sys/class/net/eth0/queues/rx-0/rps_cpusRFS config:
# Global flow table size
echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
# Per-queue flow table size
echo 2048 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt7.4 Adjust File Descriptor Limits
Edit /etc/security/limits.conf:
* soft nofile 65536
* hard nofile 65536
* soft nproc 32768
* hard nproc 32768Edit /etc/sysctl.conf:
# System-wide file descriptor limit
fs.file-max = 6553560Apply: sysctl -p. Re-login for limits.conf.
7.5 Adjust Kernel Parameters
Edit /etc/sysctl.conf:
# Increase max connections
net.core.somaxconn = 32768
# Increase TCP connection queue
net.ipv4.tcp_max_syn_backlog = 8192
# Enable TCP timestamps
net.ipv4.tcp_timestamps = 1
# Enable TCP window scaling
net.ipv4.tcp_window_scaling = 1
# Reduce TIME_WAIT connections
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
# Increase local port range
net.ipv4.ip_local_port_range = 10000 65000Apply:
sysctl -p8. Log and Metric Observation Methods
8.1 System Logs
# Last 50 lines of system log
journalctl -xe | tail -50
# Kernel log
dmesg | tail -50
# Follow system log
journalctl -fWatch keywords: Out of memory: memory exhaustion I/O error: disk I/O error segfault: process segmentation fault hung task: process hang soft lockup: soft lockup hard lockup: hard lockup
8.2 Application Logs
Common paths:
Nginx: /var/log/nginx/error.log Apache: /var/log/httpd/error_log MySQL: /var/log/mysql/error.log Redis: /var/log/redis/redis-server.log PHP-FPM:
/var/log/php-fpm/error.log8.3 Monitoring Metrics
If using Prometheus + Grafana, watch: node_load1, node_load5, node_load15: Load Average node_cpu_seconds_total: CPU time node_memory_MemAvailable_bytes: available memory node_memory_SwapCached_bytes: swap usage node_disk_io_time_seconds_total: disk I/O time node_disk_read_bytes_total: disk read bytes node_disk_written_bytes_total: disk write bytes node_network_receive_bytes_total: network receive bytes node_network_transmit_bytes_total: network transmit bytes
Metric names depend on exporter version.
9. Troubleshooting Paths
Complete paths for typical scenarios.
Scenario 1: High Load Average with High CPU Usage
Step 1: uptime confirm high load
Step 2: top confirm high CPU usage
Step 3: ps aux --sort=-%cpu find top CPU processes
Step 4: lsof -p <PID> view open files and connections
Step 5: strace -p <PID> -c view syscalls
Step 6: Analyze process behavior, confirm normalcy
Step 7: If compute-intensive, optimize code or scale
Step 8: If abnormal process, consider restart or kill
Step 9: Verify load recoveryScenario 2: High Load Average but CPU Idle
Step 1: uptime confirm high load
Step 2: top confirm high CPU idle but wa high
Step 3: iostat -x 1 5 confirm disk I/O bottleneck
Step 4: iotop -o find top I/O processes
Step 5: lsof -p <PID> view accessed files
Step 6: Check for large file ops, backups
Step 7: If excessive log writes, adjust log level or rotation
Step 8: If DB slow queries, optimize SQL or add indexes
Step 9: Verify load recoveryScenario 3: Multiple D State Processes
Step 1: ps aux | awk '$8 ~ /D/ {print $0}' find D state processes
Step 2: cat /proc/<PID>/stack view process stack
Step 3: Confirm which syscall process stuck on
Step 4: If disk I/O, check disk health
Step 5: If NFS mount, check network and NFS service
Step 6: Try force unmount or restart related services
Step 7: If unresolved, may need system rebootScenario 4: High Soft Interrupts
Step 1: top confirm high si percentage
Step 2: cat /proc/softirqs confirm network soft interrupts
Step 3: sar -n DEV 1 5 confirm high network traffic
Step 4: netstat -antp or ss -antp view connections
Step 5: Identify top network-consuming processes
Step 6: ethtool -l eth0 check NIC queue config
Step 7: Enable NIC multi-queue, RPS, RFS
Step 8: Verify soft interrupt reductionScenario 5: Memory Shortage with Frequent Swap
Step 1: free -h confirm low memory, high swap usage
Step 2: vmstat 1 5 confirm frequent si and so
Step 3: ps aux --sort=-%mem find top memory processes
Step 4: Analyze process memory usage reasonableness
Step 5: Optimize app memory or expand memory
Step 6: Temporarily clear caches (caution)
Step 7: Adjust swappiness parameter
Step 8: Verify swap usage reduction10. Risk Warnings
10.1 Cache Clearing Risks
echo 3 > /proc/sys/vm/drop_cachesclears page cache, dentries, inodes, may cause:
Subsequent file I/O slows (re-read from disk)
Ongoing I/O operations affected
Database performance temporarily degrades
Recommendations: avoid in production; only when memory critically low and cache confirmed excessive; execute during low traffic.
10.2 Force Kill Process Risks
kill -9 <PID>may cause:
Data loss or corruption
Lock files not cleaned
Child processes become orphans
Service fails to restart cleanly
Recommendations: prefer kill <PID> for graceful exit; confirm business impact and restart plan; check backups.
10.3 Force Unmount Filesystem Risks
umount -for umount -l may cause:
Processes accessing filesystem crash
Data loss
Filesystem corruption
Recommendations: confirm no processes using filesystem; use lsof /mnt/path or fuser -m /mnt/path; stop related services first.
10.4 Kernel Parameter Adjustment Risks
Wrong /etc/sysctl.conf parameters may cause:
System instability
Network anomalies
Performance degradation
Recommendations: test in staging first; backup original config; adjust one parameter at a time; document reason and expected effect.
10.5 Service Restart Risks
Restarting services may cause:
Business interruption
User request failures
Data loss
Recommendations: assess business impact first; choose low-traffic window; notify stakeholders; prepare rollback plan; monitor service post-restart.
11. Verification Methods
11.1 Verify Load Recovery
uptime
# Observe Load Average drop to normal rangeObserve continuously for minutes to confirm sustained drop or stability.
11.2 Verify CPU Usage Normal
top
# Observe %Cpu(s) id value rise
# Observe wa, si values drop11.3 Verify Memory Improvement
free -h
# Observe available memory increase
# Observe swap usage decrease11.4 Verify Disk I/O Relief
iostat -x 1 5
# Observe %util drop
# Observe await drop11.5 Verify Network Traffic Normal
sar -n DEV 1 5
# Observe rxpck/s, txpck/s return to normal levels11.6 Verify Soft Interrupt Reduction
top
# Observe %Cpu(s) si value drop11.7 Verify Process States Normal
ps -eo stat | sort | uniq -c | sort -rn
# Observe D state process reduction
# Observe R state processes in reasonable range11.8 Verify Business Recovery
Check application logs for errors
Check user feedback normalization
Check monitoring dashboards
Perform functional business validation
12. Rollback Plans
12.1 Kernel Parameter Rollback
If /etc/sysctl.conf changes cause issues:
# Restore backup config
cp /etc/sysctl.conf.bak /etc/sysctl.conf
# Apply
sysctl -p
# Or revert single parameter
# Restore swappiness default
echo 60 > /proc/sys/vm/swappiness12.2 NIC Config Rollback
If NIC queue or RPS changes cause network issues:
# Restore default queue count
ethtool -L eth0 combined 1
# Clear RPS config
echo 0 > /sys/class/net/eth0/queues/rx-0/rps_cpus12.3 Service Restart Rollback
If service restart worsens problem:
Roll back to previous stable version via version management
Restore old config from backup
Restore data from backup
Consider snapshot restore if available
12.4 Process Kill Rollback
If killing process impacts business:
# Restart service
systemctl start <service_name>
# Or manual start
/path/to/binary &Confirm after restart: process running, logs error-free, business functional.
13. Production Environment Considerations
13.1 Pre-Operation Checks
Confirm not business peak hours
Confirm no critical tasks running
Confirm backups exist
Confirm rollback plan feasible
Confirm relevant personnel notified
13.2 During Operation Monitoring
Continuously watch monitoring dashboards
Continuously watch application logs
Continuously watch system logs
Be ready to rollback anytime
13.3 Post-Operation Verification
Verify load recovery
Verify business normal
Verify monitoring normal
Document operation process and results
13.4 Permission Control
Production operations require approval
Critical operations need two-person confirmation
Record operation logs
Avoid direct root usage
13.5 Change Management
Follow change process
Record change content
Record change reason
Record change impact
Record rollback plan
13.6 Emergency Preparedness
Maintain emergency contact list
Prepare emergency handling procedures
Prepare common command checklist
Regularly drill emergency processes
14. Summary
Troubleshooting high Linux system load is a fundamental skill for operations engineers, requiring mastery of:
14.1 Understand Core Concepts
Load Average reflects runnable and uninterruptible sleep process counts
Must combine with CPU core count to judge normality
High Load Average ≠ high CPU; could be I/O bottleneck
14.2 Establish Troubleshooting Path
Overall first, then local
Observe first, then act
Eliminate obvious issues first
Non-intrusive first, then intrusive
Record the scene
14.3 Master Tools
uptime, top, htop: load and CPU free, vmstat: memory and swap iostat, iotop: disk I/O sar, netstat, ss: network ps, lsof, strace, pstack: process analysis
14.4 Master Analysis Methods
Categorize by CPU, memory, disk, network
Focus on process state distribution, especially R and D
Deep-dive into abnormal process syscalls, open files, network connections
Correlate multiple metrics to determine root cause
14.5 Apply Effective Measures
Temporary: lower priority, clear caches, throttle
Root: optimize code, adjust config, scale resources
High-risk ops require impact assessment and rollback prep
Production ops must follow change process
14.6 Close the Loop
Must verify effectiveness after fix
Prepare rollback plan
Document troubleshooting process and results
Produce incident reports and knowledge base
Optimize monitoring alerts and prevention mechanisms
Only through extensive practical accumulation can one quickly locate issues, take correct actions, and restore business during production incidents. This article provides a systematic troubleshooting methodology and toolset, to be flexibly applied per specific business scenarios and system characteristics.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
