Essential Linux System Monitoring and Troubleshooting Commands

View Logs

1. Show logs below warning level for today:

man journalctl | grep -C 2 "debug" | journalctl -xen all --since today -p warning [-o json | -o json-pretty]

2. System boot and kernel logs:

# Boot logs
journalctl -b [-0]
# Kernel logs
journalctl -k

3. Latest logs for a specific service or process:

# Service logs
journalctl -xen 10 -fu sshd
# Process logs
ss -tnlp | grep -P '(?<=pid=)\d+'
journalctl -xen 10 _PID=1 -f
journalctl /usr/bin/bash

4. Manage journal size:

# Disk usage
journalctl --disk-usage
# Keep only 500M
journalctl --vacuum-size 500M
# Keep logs for 1 year
journalctl --vacuum-time 1years

Inspection Commands

CPU

// CPU usage
 top
// CPU core info
 mpstat -P ALL 1
// CPU usage and load average
 vmstat 1 3
// Detailed vmstat output
 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 1379800   2188 2044368    0    0     0     2    1    1  0  0 100  0  0
 // ... (output truncated for brevity)

Additional CPU tools: pidstat -u 1 -p pid, perf top -p pid -e cpu-clock. Use uptime and w to view load averages and sar -q for dynamic load.

Memory

// Memory usage
 free -m
// Virtual memory stats
 vmstat 1
// Detailed memory info
 top
// Per‑process memory stats (1‑second interval, 5 samples)
 pidstat -p pid -r 1 5
// Process memory map
 pmap -d pid
// Detect memory leaks
 valgrind --tool=memcheck --leak-check=full --log-file=./log.txt ./program

Disk I/O

// Real‑time I/O view
 iotop
// Detailed I/O statistics
 iostat -d -x -k 1 10
// Per‑process I/O
 pidstat -d 1 -p pid
// Trace block requests
 perf record -e block:block_rq_issue -a
 perf report

Network

// Network statistics
 netstat -s
// UDP connections
 netstat -nu
// UDP port usage
 netstat -apu
// TCP connection counts by state
 netstat -a | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
// TCP connections
 ss -t -a
// Socket summary
 ss -s
// UDP sockets
 ss -u -a
// TCP/ETCP stats
 sar -n TCP,ETCP 1
// Network I/O stats
 sar -n DEV 1
// Packet capture
 tcpdump -i eth1 host 192.168.1.1 and port 80
// Flow capture
 tcpflow -cp host 192.168.1.1

System Load

// Load overview
 uptime
 top
 vmstat
// System call time profiling
 strace -c -p pid
// Trace specific syscalls (e.g., epoll_wait)
 strace -T -e epoll_wait -p pid
// Kernel log
 dmesg

Common Commands

1. Temporarily configure host IP:

ifconfig eth0 192.168.1.1
ip a add 192.168.1.100/24 dev eth0   # set IP and netmask
ip r add default via 192.168.1.100 dev eth0   # default gateway
ip r add 0.0.0.0/0 via 192.168.1.254 dev eth0   # add route

2. Convert Windows line endings to Linux: sed -i -e 's/\r$//' install.sh 3. Query public IP address:

curl cip.cc
curl ipinfo.io
curl ip.cn
curl myip.ipip.net
curl ifconfig.me
curl ip.sb
curl inet-ip.info
curl ident.me
curl icanhazip.com
curl myip.ipip.net

4. Continuous service availability testing:

while true; do curl 172.31.76.10/service/colors; sleep 0.$RANDOM; done
while true; do curl 172.31.18.2/livez && echo -e "\t" && curl 172.31.18.2; sleep 1; done
while true; do curl 172.31.29.2; sleep .5; done

5. Find large files or directories (>200 MB):

find / -type f -size +200M -print0 | du --files0-from=- --max-depth=1 -h | sort -hr

6. Locate high‑resource processes:

# Top CPU consumers
ps -eo pid,ppid,user,%cpu,%mem,cmd --sort=-%cpu | head -10
# Top memory consumers
ps -eo pid,ppid,user,%cpu,%mem,cmd --sort=-%mem | head -10
# Top swap consumers (use top, Shift+F, select swap, sort)

# Process details
ps -p <PID> -o comm,cmd
# Service owning the process
cat /proc/<PID>/cmdline
# Open files
lsof -p <PID>
# Number of open files
ls -l /proc/<PID>/fd/ | wc -l

7. Release cached memory manually:

# Sync and drop caches
sync
echo 3 > /proc/sys/vm/drop_caches