Essential Linux Ops Commands: 50+ Tools to Cut Troubleshooting Time in Half

Problem Background

Many junior ops engineers only know basic commands like top, df -h, free -m and struggle when facing unfamiliar output, causing troubleshooting to take over an hour. The core competency is not how many commands you know, but being able to map a symptom to the right command within 5 seconds — a reflex built through extensive practice.

The author, with 8 years of ops experience, compiled 50+ daily troubleshooting commands organized by theme, each with purpose, key parameters, output interpretation, practical snippets, and caveats. After studying this guide, you should instantly know which commands to run for scenarios like high load average, slow business, disk full, inode exhaustion, or stuck processes.

Applicable Scenarios

Physical/virtual/cloud server daily operations

Application service (Nginx, Tomcat, MySQL, Redis, Kafka) process-level investigation

CPU, memory, disk I/O, network performance analysis

Process state, system calls, kernel event tracing

Text processing, batch operations, automation scripts

Fault localization, root cause analysis, performance tuning

Not covered: macOS (BSD-style), Windows Server, container-specific commands (resource limits, namespaces differ), large distributed systems (Hadoop, Ceph, Kafka clusters).

Core Knowledge

3.1 Linux Command "Reflex Arc"

First reflex: symptom → command. Typical mappings:

Business slow: uptime / top / vmstat Memory pressure: free -h / dmesg High disk I/O: iostat -xz 2 / iotop Network unreachable: ss -lntp / mtr / tcpdump Process disappeared: systemctl status / journalctl -u Disk full: df -h / du -sh /* Inode full: df -i System logs: journalctl -p err System calls: strace -p <pid> Files opened by process:

lsof -p <pid>

3.2 Command Risk Levels

Safe (read-only): uptime, free, df, ss -s, ps Observation (may affect performance): strace, tcpdump, perf top Modification (explicit state change): systemctl restart, kill, iptables -I Destructive (high risk): rm -rf, dd, mkfs, fdisk Always classify a command before execution. Destructive commands require backup, approval, and rollback plan.

3.3 Key Directory Quick Reference

/proc/<pid>/

— Process runtime info (memory, files, state) /proc/sys/ — Kernel runtime parameters /proc/loadavg — System load /proc/meminfo — Memory details /sys/fs/cgroup/ — cgroup resource limits /var/log/ — System logs /etc/systemd/system/ — systemd unit configs /sys/class/net/ — Network interface runtime info

Knowing these directories lets you troubleshoot even when standard tools are unavailable.

Overall Command Organization

Status Query: View system/process current state

Problem Location: Trace root cause from state

Change Execution: Modify config or restart service

Batch Processing: Handle multiple files/processes

Information Extraction: Extract key info from logs/output

Process Commands

5.1 ps: Process Snapshot

Purpose: List all process states.

Key Parameters:

ps aux                  # BSD style, all processes
ps -ef                  # System V style
ps -eo pid,ppid,user,stat,pcpu,pmem,etime,comm,args --sort=-pcpu

Output Interpretation: USER (owner), PID, %CPU, %MEM, STAT (R/S/D/Z/T/I), START, TIME, COMMAND.

Practical Snippets:

# Top 10 CPU consumers
ps -eo pid,ppid,user,pcpu,pmem,comm --sort=-pcpu | head -11
# Top 5 memory consumers
ps -eo pid,user,pmem,comm --sort=-pmem | head -6
# Find zombie processes
ps -eo pid,ppid,stat,comm | awk '$3 ~ /Z/'
# Processes running >24h
ps -eo pid,etime,comm | awk '$2 ~ /-/ {print}'
# Full command line
ps -ef | grep <name> | grep -v grep

Caveats: ps aux column order varies by distro; STAT D = uninterruptible sleep (I/O block); etime shows elapsed time (e.g., 5-01:23:45); Linux 3.x+ requires a or x to show all processes.

5.2 top: Real-time Process Monitoring

Purpose: Dynamic system/process view.

Key Parameters:

top -b -n1              # Batch mode, single run (scripts)
top -p <pid1>,<pid2>    # Monitor specific PIDs
top -u <user>           # Specific user
top -H -p <pid>         # Show threads

Interactive Keys: P (CPU sort), M (memory sort), T (cumulative CPU time), 1 (per-core), H (threads), c (full command), k (kill), q (quit).

Practical Snippets:

top -b -n1 -o %CPU | head -20
# Java threads
top -H -p <pid>
# Specific user
top -u www-data

Caveats: Avoid accidental k in interactive mode; top -b -n1 slower than ps but shows real-time CPU; prefer htop for better UI and mouse support.

5.3 htop: Enhanced top

Install: yum/apt install htop. Usage: htop -d 5 (5s refresh), htop -u user, htop -p PID, htop -t (tree view). Mouse-clickable, color-codes D (red) and R (green), drag for tree.

5.4 pidstat: Per-process Resource Stats

Purpose: Sample process CPU, memory, I/O at intervals.

Key Parameters:

pidstat -p <pid> 2 5    # Every 2s, 5 samples
pidstat -u 2 5          # All processes CPU
pidstat -r 2 5          # Memory
pidstat -d 2 5          # I/O
pidstat -w 2 5          # Context switches

Practical Snippets:

# Monitor Java CPU
pidstat -p <pid> 2 5 -u
# MySQL I/O
pidstat -p $(pidof mysqld) 2 5 -d
# Highest context switches
pidstat -w 2 5 | head -20

Caveats: Part of sysstat package (CentOS 7+/Ubuntu 16+); better for scripting than top; pidstat -t shows thread-level.

5.5 pstree: Process Tree

Shows parent-child relationships. pstree -a (command line), -p (PID), -u (user changes), -s <pid> (parent chain). Use less for long output.

5.6 strace: System Call Tracing

Purpose: Trace syscalls and signals.

Key Parameters:

strace -p <pid>                 # Attach to running process
strace -c -p <pid>              # Summary stats
strace -e trace=network,read,write -p <pid>  # Filter
strace -f -p <pid>              # Follow children
strace -T -p <pid>              # Show syscall duration
strace -tt -o /tmp/trace.log <cmd>  # Start and trace

Practical Snippets:

# Java process activity
strace -p <pid> -T -tt -e trace=network,read,write
# Syscall distribution
strace -c -p <pid>
# Trace MySQL startup
strace -f -o /tmp/mysql-strace.log mysqld --user=mysql

Caveats: 5-30% performance hit — avoid on production high-load; output large, use tee; needs ptrace (containers lack by default); must be root if target is root; -c prints summary on Ctrl+C.

5.7 lsof: List Open Files

Purpose: View process-open files, sockets, pipes.

Key Parameters:

lsof -p <pid>           # All files for PID
lsof -i :80             # Process on port 80
lsof /var/log/messages  # Who writes this file
lsof -u <user>          # User's open files
lsof +D /var/log        # Recursive directory
lsof -i -P -n           # TCP/UDP connections

Practical Snippets:

lsof -i :80
lsof /var/log/messages
lsof -p <pid> -i -P -n
lsof +D /var/log | grep deleted

Caveats: Output large, pipe to head / grep; +D recursive slow on big dirs; lsof -i more detailed than ss -lntp.

5.8 /proc/PID/*: Process Runtime Info

Direct filesystem access to process data:

/proc/<pid>/cmdline      # Command line (NUL-separated)
/proc/<pid>/cwd          # Current working directory (symlink)
/proc/<pid>/environ      # Environment variables
/proc/<pid>/fd/          # Open file descriptors
/proc/<pid>/maps         # Memory mappings
/proc/<pid>/status       # Process status
/proc/<pid>/stat         # Process statistics
/proc/<pid>/statm        # Memory stats
/proc/<pid>/wchan        # Kernel function where sleeping
/proc/<pid>/limits       # Resource limits

Practical Snippets:

# Command line (convert NUL)
cat /proc/<pid>/cmdline | tr '\0' ' '; echo
# Environment variables
cat /proc/<pid>/environ | tr '\0' '
'
# Open FDs
ls -la /proc/<pid>/fd/ | head
# Kernel sleep function (when stuck)
cat /proc/<pid>/wchan; echo
# Status
cat /proc/<pid>/status
# Memory map
cat /proc/<pid>/maps | head
# Limits
cat /proc/<pid>/limits

Caveats: /proc/PID/fd/ are symlinks; files disappear after process dies; wchan often empty unless root.

Memory Commands

6.1 free: Memory Usage

Purpose: Quick view of total, used, free, available, cache.

Key Parameters: free -h (human), -m (MB), -g (GB), -s 2 (refresh 2s), -c 5 (5 times).

Output Fields: total, used (includes buff/cache), free, shared, buff/cache (reclaimable), available (free + reclaimable cache).

Practical Snippets:

# Key metric is available, not free
free -h
# If available near 0 and used near total, memory truly low
watch -n 2 free -h

Caveats: Newbies mistake low free for shortage; buff/cache is page cache, reclaimable; available is true allocatable memory.

6.2 /proc/meminfo: Memory Details

Source for free. Key metrics:

grep -E 'MemTotal|MemFree|MemAvailable|Buffers|Cached|SwapTotal|SwapFree|Dirty|Writeback|AnonPages|Mapped|Shmem' /proc/meminfo

Watch Dirty (pages waiting for disk) — growth indicates I/O bottleneck; AnonPages high = program heap/stack usage.

6.3 pmap: Process Memory Map

Purpose: Process memory distribution.

Key Parameters: pmap -x PID (detailed), -d (device format), pmap PID | sort -k2 -nr | head (sort by RSS).

Practical Snippets:

pmap -x $(pidof mysqld) | sort -k3 -nr | head -20
pmap -x <pid> | awk 'NR>1 {print $2, $4}' | sort -k2 -nr | head -10

Caveats: Output long, sort and head; RSS = resident, Size = virtual; friendlier than /proc/PID/maps.

6.4 vmstat: Virtual Memory Stats

Purpose: Combined CPU, memory, I/O, context switches.

Key Parameters: vmstat 2 5, vmstat -s (one-time stats), vmstat -d (disk stats).

Key Fields: r (runnable processes, >CPU cores = CPU saturation), b (uninterruptible sleep = I/O block), swpd (swap used), si/so (swap in/out), bi/bo (block I/O), in (interrupts), cs (context switches), us/sy/id/wa/st (user/sys/idle/iowait/steal).

Practical Snippets:

vmstat 2 5
# wa >30% = severe I/O
# cs >100k/s = potential issue
# st >10% (VM) = host contention

Caveats: First line is boot average — ignore; r > CPU cores = CPU bottleneck; b >0 = I/O block; si/so >0 = physical memory shortage.

6.5 slabtop: Kernel Slab Cache

View kernel slab allocator usage. slabtop -d 2 real-time; slabtop | head -20 for dentry cache; cat /proc/slabinfo | head -20 for details. Persistent growth may indicate kernel memory leak; dentry growth = many open files.

CPU Commands

7.1 uptime: System Load

Shows 1/5/15 min load averages. Healthy load ≤ CPU cores; 1m ≫ 5m/15m = recent spike; all high = sustained issue. Load high ≠ CPU full — check iowait. Rule: load < cores = healthy, >2× cores = severe. Use nproc for core count.

7.2 mpstat: Per-core CPU Stats

mpstat -P ALL 2 5

(all cores), mpstat -I ALL 2 5 (interrupts), mpstat -I CPU 2 5 (interrupts per CPU). High %usr = user process; %sys = kernel/syscalls; %iowait = I/O block; %steal (VM) = host contention; %soft = softirq storm.

7.3 sar: System Activity Reporter

Long-term sampling, historical replay. sar -u CPU, -r memory, -d disk, -n DEV network, -b I/O rate, -q queue length. Historical: sar -u -f /var/log/sa/sa01 (yesterday). Requires sysstat package and config; retains 7 days by default. Best for "yesterday's fault" analysis.

7.4 perf: Linux Performance Analysis

Kernel/app profiling. perf top real-time hotspots; perf top -p PID; perf record -p PID -g -- sleep 30 + perf report; perf stat -e context-switches,cs,migrations -p PID sleep 5. Needs linux-tools; supports tracepoints/kprobes; perf top heavy on high-load; containers need --privileged or perf_event_open.

Disk I/O Commands

8.1 iostat: Disk I/O Stats

Purpose: Per-disk read/write rates, utilization, wait times.

Key Parameters: iostat -xz 2 5 (extended, skip zero), -m (MB), -d sda (specific disk), -p ALL (partitions).

Key Fields: r/s, w/s (IOPS), rkB/s, wkB/s (throughput), await (avg wait ms = queue + service), svctm (service time, deprecated), %util (device saturation).

Practical Snippets:

iostat -xz 2 5
# %util >60% busy
await >10ms (HDD) or >1ms (SSD) = problem

Caveats: %util near 100% = saturated; await spikes with low %util may indicate RAID/SAN issues; first line is boot average; %util less meaningful on SSD — focus on await.

8.2 iotop: Per-process I/O

Find top I/O consumers. iotop -o (only active I/O), -b -n 3 (batch), -a (accumulated), -p PID. Requires iotop package; columns: disk read, write, IO%, command; containers need SYS_ADMIN.

8.3 smartctl: Disk Health

SMART info for failure prediction. smartctl -a /dev/sda (all), -H (health), -t long (long self-test), -l selftest (log). Key: Reallocated_Sector_Ct growth = impending failure. Physical only; self-tests degrade performance.

Filesystem Commands

9.1 df: Disk Space

df -h

(human), -hT (with FS type), -i (inodes), -h /path (partition for path). Must check both space and inodes — inode exhaustion also blocks writes. df -hT shows ext4/xfs/tmpfs; bind mounts counted with source.

9.2 du: Directory Size

du -sh /path

(summary), -h --max-depth=1 (first level), du -ah | sort -h | tail (largest files). Slow on large dirs — use ionice -c3 nice -n 19 du ... on busy systems.

9.3 ls: File Listing

ls -lh

(human), -lhS (size sort), -lht (time sort), -lR (recursive), -la (hidden). Slow on huge dirs — ls -U skips sort; ls -d */ only directories.

9.4 mount: Mount Points

mount | column -t

for aligned view. Filesystem turned read-only ( ro,relatime) usually indicates hardware issue. mount -o remount,ro / can force read-only.

9.5 tune2fs / xfs_info: FS Parameters

ext4: tune2fs -l /dev/sda1; xfs: xfs_info /dev/sda1 or xfs_info /mnt/data. Backup before modifying FS params.

Network Commands

10.1 ss: Socket Statistics (modern netstat)

ss -lntp

(listening TCP), -anp (all), -s (summary), -tan state established, state time-wait, -i (details). Faster than netstat. TIME_WAIT pileup = many short connections; CLOSE_WAIT pileup = app bug (not closing). Need root for process names.

10.2 netstat: Legacy

May be absent on newer distros (CentOS 8+, Ubuntu 22+). Prefer ss.

10.3 ip: Network Config (modern ifconfig)

ip a

(addresses), ip route (routing), ip neigh (ARP), ip link (interfaces), ip -s link (stats). ip route get 8.8.8.8 shows route to target. Changes immediate, not persistent.

10.4 mtr: Network Diagnostics

Combines ping + traceroute. mtr -rwn -c 20 target (report, 20 pings). Loss% = packet loss, Avg = avg latency. Intermediate node loss not necessarily that node's fault — next hop loss confirms. Needs root/CAP_NET_RAW.

10.5 traceroute: Route Tracing

traceroute -T -n -w 2 -m 15 target

(TCP SYN, more reliable). UDP often blocked; * * * = node not responding (ICMP disabled).

10.6 tcpdump: Packet Capture

tcpdump -i eth0 -nn -A -s 0 port 80

(HTTP), port 3306 -w /tmp/mysql.pcap (MySQL), 'tcp[tcpflags] & tcp-syn != 0' (SYN packets), port 53 (DNS). Use -s 0 for full packets; high-traffic interfaces may drop; analyze pcap with Wireshark; limit with -c to control disk usage.

10.7 dig: DNS Query

dig +short example.com

(concise), +trace (full resolution), @8.8.8.8 (specific resolver), -x (reverse). More detailed than nslookup.

10.8 nslookup / 10.9 host: DNS Alternatives

nslookup deprecated in some distros; prefer dig or host. host example.com, host -t mx example.com.

Log & Package Commands

11.1 dmesg: Kernel Ring Buffer

dmesg -T

(human time), -T -l err (errors only), -T -f kern (kernel only), -C (clear — dangerous). Search OOM: dmesg -T | grep -i -E 'oom|killed'; hardware errors: grep -i -E 'mce|machine check|hardware error'; mount errors: grep -i 'mount'. Logs persist to /var/log/kern.log or messages after reboot.

11.2 journalctl: systemd Logs

journalctl -u nginx.service

, --since '1 hour ago', -p err -b (this boot errors), -f (follow), _PID=PID, -k (kernel), --vacuum-time=7d (cleanup). Default in-memory (lost on reboot) unless persistent storage configured at /var/log/journal/. Use --no-pager for scripts.

11.3 rpm/yum: CentOS/RHEL Package Mgmt

rpm -qa --last | head

(recent installs), rpm -V pkg (verify files), rpm -qf /path/file (file owner), rpm -ql pkg (package files), yum history (install history), yum history info ID, yum history undo ID (rollback). rpm -V anomalies may indicate intrusion; yum history undo may fail on dependencies.

11.4 dpkg/apt: Debian/Ubuntu Package Mgmt

dpkg -l

, -L pkg, -S /path/file, -V (verify), debsums -c (needs install), apt list --installed, apt log. dpkg -V may complain about logrotate-removed files.

Text Processing Trio

12.1 grep: Search

Key flags: -n (line numbers), -i (ignore case), -r (recursive), -E (extended regex), -v (invert), -c (count), -l (filenames only), -A/B/C 5 (context), --color. Use -F for literal strings, --mmap for large files.

12.2 awk: Processing

awk '{print $1}'

, -F: (delimiter), '$3 > 80' (condition), 'NR>1' (skip header), '{sum+=$1} END {print sum}' (sum), -v t=10 '$1 > t' (variable). $0=whole line, $1,$2=fields, NR=line number, NF=field count. Powerful regex replaces many grep+sed combos.

12.3 sed: Stream Editor

s/old/new/

(first per line), g (all), -i.bak (in-place with backup), -n '5,10p' (lines 5-10), '/^#/d' (delete comments), '/pattern/d' (delete matches). Always backup with .bak; change delimiter if path contains / (use # or |).

12.4 xargs: Argument Builder

-I {} cmd {}

(placeholder), -n 1 (one arg), -P 4 (4 parallel), -d '\n' (delimiter). Without -I, splits on whitespace — breaks on spaces. Use find -print0 | xargs -0 for special chars.

find: File Search

Key: -name '*.log', -type f/d, -size +100M, -mtime -7 (modified within 7 days), +30 (older than 30 days), -user www-data, -perm 777, -delete (dangerous). Always dry-run with -print first; -xdev avoids crossing filesystems (/proc, /sys); limit depth with -maxdepth; never find / -delete.

Service Management

14.1 systemctl

status/start/stop/restart/reload/enable/disable

, list-units --type=service, --state=failed, cat service (unit file), show service (runtime props). reload no downtime (if supported); restart brief interruption; inactive (dead) may be normal; active (running) but port not listening = false positive.

14.2 journalctl

See 11.2.

14.3 service: SysV Compat

Calls systemctl; only for compatibility.

File & Permission Commands

15.1 stat: File Metadata

Shows Access/Modify/Change times. Access time may be inaccurate on 2.6+; Modify = content change; Change = metadata change.

15.2 getfacl/setfacl: ACLs

getfacl /path

, setfacl -m u:nginx:r-x /path, setfacl -x u:nginx /path. FS must be mounted with acl option; more flexible than ugo; backup with getfacl -R.

15.3 chattr/lsattr: File Attributes

chattr +i /etc/passwd

(immutable), chattr +a /var/log/messages (append-only). Root can still remove +i; +a allows only append; emergency removal with chattr -i.

Resource Limits

16.1 ulimit: Process Limits

ulimit -a

(all), -n (file descriptors), -u (processes), -s (stack), -t (CPU time), -v (virtual memory). Temporary: ulimit -n 65535; permanent: add to /etc/profile. Effective in login shell; systemd services use LimitNOFILE= in unit file.

16.2 /etc/security/limits.conf

Format: domain type item value. Example: * soft nofile 65535, * hard nofile 65535, root soft nofile 65535, root hard nofile 65535. Requires re-login; systemd services ignore this — set in unit.

Configuration Examples

17.1 Process Info Collection Script

Bash script taking PID, outputs ps, /proc/PID/status, wchan, open FDs, socket count.

17.2 System Health Check Script

Runs uptime, mpstat, free, df, df -i, iostat, ss, systemctl --failed.

Risk Warnings

rm -rf

— always ls first find -delete — always find -print first kill -9 on DB causes corruption iptables -F — lose connectivity chmod 777 — dangerous master key chattr +i — file becomes unmodifiable dd if=/dev/zero of=/dev/sda — wipes disk

Backup configs before changes

Quote paths in scripts

Verify network stability before remote ops

Verification Methods

Run health-check.sh and inspect output

Load test with wrk for comparison

Check monitoring baselines

Use strace -c for syscall distribution

Write runbooks for junior execution

Rollback Strategies

cp

config before edit systemctl show before service changes iptables-save before firewall changes

Keep configs in Git

Backup DB schema before changes

Record yum history ID before upgrades

Production Best Practices

Think thrice before destructive ops

Critical ops need a second person aware

No high-risk changes during peak hours

Test config changes in staging first

Monitoring first: baseline then change

Important changes go through change tickets

Regular backups of DB/etcd

Document command risk levels in runbooks

Verify container namespaces

Secrets via env vars or vault, not scripts

Summary

Internalizing 50+ commands as muscle memory is fundamental for ops engineers. Coverage spans CPU, memory, disk I/O, network, processes, logs, services, text processing, resource limits — all common troubleshooting scenarios.

Advice for junior/mid-level engineers:

Type every command yourself: reading ≠ knowing

Build symptom→command reflex: observe first, then choose tool

Take notes: record real-world scenarios for each command

Create runbooks: organize commands into executable docs per scenario

Risk awareness: classify command risk before execution

Continuous learning: track new tools (bpftrace, systemd-cgtop)

Combine with monitoring: commands + alerts = fast localization

Postmortem: after each incident, document commands used

With these 50+ commands mastered, plus monitoring/alerting and runbooks, you can go from "business is slow" to "root cause identified" in 5 minutes — the rest is just executing the playbook.