10 Essential System Commands Every Ops Engineer Should Master

1. Command Classification

The ten essential Linux commands are grouped into five categories:

System monitoring : top, htop, vmstat, iostat Network diagnostics : netstat, ss, tcpdump, lsof Disk analysis : df, du Text processing : find, awk, sed Service management : systemctl, service,

rsync

2. Detailed Command Walk‑throughs

2.1 top and htop

top

shows real‑time process and resource usage. The first line displays load averages; values higher than the number of CPU cores indicate a bottleneck. Interactive keys: P – sort by CPU usage M – sort by memory usage T – sort by runtime 1 – show per‑CPU statistics c – display full command line k – kill a process q – quit

Example output:

top - 10:15:30 up 100 days, 3:22, 2 users, load average: 1.25, 0.98, 0.85
Tasks: 245 total, 3 running, 242 sleeping, 0 stopped, 0 zombie
%Cpu(s): 25.3 us, 5.2 sy, 0.0 ni, 68.5 id, 0.0 wa, 0.8 hi, 0.2 si, 0.0 st

htop

is a colour‑enhanced, mouse‑friendly version that supports process trees and custom layouts. Common keys: F1 help, F2 setup, F3 search, F4 filter, F5 tree view, F6 sort, F9 kill, F10 quit.

2.2 netstat and ss

netstat -anp

lists all connections with protocol, local/foreign address, state, and owning process. Typical TCP states include LISTEN , ESTABLISHED , TIME_WAIT , CLOSE_WAIT .

netstat -anp
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1234/sshd

ss -tunap

provides the same information faster and with richer output.

ss -tunap
Netid  State   Recv-Q Send-Q Local Address:Port      Peer Address:Port       Process
tcp    LISTEN  0      0      0.0.0.0:22               0.0.0.0:*               users:("sshd",pid=1234,fd=3)

2.3 lsof

lsof

lists open files, which on Linux includes sockets, pipes and device files. Useful flags: -i – show network files -p <PID> – filter by process ID -u <user> – filter by user

lsof -i:80          # show process using port 80
lsof -p 1234       # list files opened by PID 1234

2.4 df and du

df -h

displays filesystem size, used, available and usage percentage. df -i shows inode consumption.

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        100G   45G   55G  45% /

du -sh /var

reports total size of a directory; du -sh * lists sizes of sub‑directories.

du -sh /var
12G    /var/log
8G     /var/lib
5G     /var/cache

2.5 iostat and vmstat

iostat -xz 1

provides per‑device I/O statistics. Key metrics: %util (device utilisation) and await (average I/O wait). High %util indicates an I/O bottleneck.

iostat -xz 1
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s await %util
sda      0.12   12.34 2.45 45.67 198.23 4567.89 5.67 5.67

vmstat 1

shows run‑queue length ( r), blocked I/O processes ( b), free memory and CPU percentages.

vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so   bi   bo   in   cs us sy id wa st
 3  0      0 2048576 312456 4567890 0   0    5   12 1234 8567 25 5 68 2 0

2.6 strace and ltrace

strace -c -p <PID>

aggregates system‑call counts and time, helping pinpoint CPU‑bound syscalls.

strace -c -p 1234
% time   seconds   usecs/call   calls   errors   syscall
45.23    0.123456   123          1000               read
25.67    0.070123   70           1000               write

ltrace -p <PID>

traces library function calls; ltrace -c -p <PID> counts them.

2.7 rsync

Incremental synchronization, transferring only changed blocks.

rsync -avz --delete source/ dest/          # local sync, delete extraneous files
rsync -avz -e "ssh -p 2222" source/ user@host:/dest/   # remote sync with custom SSH port
rsync -avzn source/ dest/                 # dry‑run to preview actions

2.8 systemctl and service

Standard systemd service management:

systemctl start nginx
systemctl status nginx
systemctl enable nginx
journalctl -u nginx -n 50   # recent logs
service nginx restart      # SysVinit compatibility

3. Practical Troubleshooting Scenarios

3.1 High Server Load

Identify the bottleneck: top or vmstat 1. If us or sy is high, the issue is CPU‑bound; if wa is high, it is I/O‑bound.

CPU‑bound: sort processes by CPU ( P in top) and investigate high‑usage processes (e.g., Java). Use jstack or pidstat -p <PID> 1 to drill down.

I/O‑bound: check iostat -xz 1 for devices with high %util. Use iotop to locate the offending process and lsof +D / to see open files.

After pinpointing the culprit, consider code optimisation, workload throttling, or hardware upgrades.

3.2 Network Connectivity Issues

Test basic reachability: ping -c 4 <target_ip> and traceroute <target_ip>.

Verify the service is listening: ss -tlnp or netstat -tlnp.

Capture traffic:

tcpdump -i any host <target_ip> and port <port> -w capture.pcap

and analyse with Wireshark.

Check firewall rules: iptables -L -n -v and SELinux/AppArmor status.

3.3 Disk Space Exhaustion

Show filesystem usage: df -h and inode usage: df -i.

Drill down: du -h --max-depth=1 / then du -sh /var/* to locate large directories.

Find large files: find /var -type f -size +100M -exec ls -lh {} \;.

Clean old logs: find /var/log -name "*.log" -mtime +30 -delete and release deleted‑but‑open files with lsof +L1.

4. Decision Tree for Command Selection

When a system feels sluggish, start with vmstat 1 5 or top. If us or sy is high, focus on CPU‑intensive processes; if wa is high, move to I/O analysis with iostat and iotop. For network slowness, jump to ss -s or netstat -an, then capture packets with tcpdump. For disk pressure, begin with df -h and follow the du hierarchy.

5. Advanced Tools and Automation

5.1 Enhanced Monitoring Tools

htop

– colour UI, process tree, per‑CPU view ( -d refresh delay, -p PID filter). iotop – real‑time per‑process I/O ( -o show only active I/O, -b batch mode). iftop – network bandwidth per connection ( -i <iface>, -n no DNS, -B bytes). nethogs – network bandwidth per process ( nethogs eth0).

5.2 Log Analysis Tools

goaccess

– real‑time web log analyser ( goaccess access.log -c). lnav – advanced log viewer ( lnav /var/log/syslog).

5.3 System Information Collection

dmidecode -t system

, -t memory, -t processor – hardware details. lscpu – CPU architecture. lsblk -f – block device layout.

5.4 Fault‑Isolation Workflow

Standardised phases:

Information collection : uname -a, uptime, df -h, free -h, top -bn1.

Network diagnosis : ping, traceroute, nslookup, netstat -tlnp, ss -tunap.

Process & service check : ps aux, pstree, systemctl status <service>, journalctl -u <service> -n 50.

Log review : tail -100 /var/log/syslog, tail -100 /var/log/nginx/error.log.

Resource analysis : vmstat 1, mpstat -P ALL 1, iostat -xz 1, free -m.

5.5 Sample Automation Scripts

System health‑check script (bash):

#!/bin/bash
LOG_FILE="/var/log/health_check.log"
EMAIL="[email protected]"

log() {
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

check_cpu() {
  cpu=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | sed 's/%us,//')
  if (( $(echo "$cpu > 80" | bc -l) )); then
    log "WARNING: CPU usage is high: $cpu%"
    return 1
  fi
  log "OK: CPU usage is normal: $cpu%"
}

check_memory() {
  mem=$(free | grep Mem | awk '{print ($3/$2) * 100}')
  if (( $(echo "$mem > 85" | bc -l) )); then
    log "WARNING: Memory usage is high: $mem%"
    return 1
  fi
  log "OK: Memory usage is normal: $mem%"
}

check_disk() {
  usage=$(df -h / | tail -1 | awk '{print $5}' | sed 's/%//')
  if [ "$usage" -gt 80 ]; then
    log "WARNING: Disk usage is high: $usage%"
    return 1
  fi
  log "OK: Disk usage is normal: $usage%"
}

check_services() {
  for svc in nginx mysql redis; do
    if systemctl is-active --quiet $svc; then
      log "OK: $svc is running"
    else
      log "ERROR: $svc is not running"
      return 1
    fi
  done
}

log "=== System Health Check Started ==="
check_cpu
check_memory
check_disk
check_services
log "=== System Health Check Completed ==="

Log backup and cleanup script (bash):

#!/bin/bash
LOG_DIR="/var/log/myapp"
BACKUP_DIR="/backup/logs"
RETENTION_DAYS=30

mkdir -p "$BACKUP_DIR"

timestamp=$(date +%Y%m%d_%H%M%S)
 tar -czf "$BACKUP_DIR/logs_${timestamp}.tar.gz" "$LOG_DIR"/*.log 2>/dev/null

find "$BACKUP_DIR" -name "logs_*.tar.gz" -mtime +$RETENTION_DAYS -delete
find "$LOG_DIR" -name "*.log" -mtime +7 -exec gzip {} \;
find "$LOG_DIR" -name "*.log.gz" -mtime +90 -delete

echo "Log backup and cleanup completed at $(date)"

These commands, examples and scripts provide a concrete, step‑by‑step toolkit for monitoring, diagnosing and automating routine Linux‑server operations.