Boost Ops Efficiency: 10 Essential Linux Tools Every Engineer Should Use

Background and Use Cases

Linux ships with many useful command‑line tools, but many engineers only use the most basic ones and miss hidden gems. This guide targets junior to mid‑level operations engineers, system administrators, and DevOps practitioners, focusing on the ten most frequently used, high‑impact tools. Each tool is presented with a realistic scenario, core usage, advanced options, and a concrete troubleshooting case.

Selection Criteria for Tools

Production‑validated stability

Native Linux or mainstream open‑source

Solves real operations pain points

No reliance on commercial closed‑source software

Tool 1: htop / atop – Interactive Process Monitoring

Scenario

CPU usage spikes on a production server. top shows many processes but locating the culprit is cumbersome.

Installation

# RHEL/CentOS
yum install htop atop -y

# Debian/Ubuntu
apt install htop atop -y

Core Usage

htop basics

# Start htop
htop

# Common shortcuts (press the key while htop is running)
# ↑↓   select process
# Enter view process details
# Space tag/untag process
# u     show processes of a specific user
# P     sort by CPU (Shift+P locks)
# M     sort by memory
# T     sort by runtime
# F     toggle sort field
# k     send signal (TERM then KILL if needed)
# /     search process name
# F4    filter processes
# k + 15 send TERM first, then SIGKILL

htop advanced

# Run as a specific user
htop -u mysql

# Start in tree view
htop -t

# Show specific PIDs
htop -p 12345,12346,12347

# Show full command line
htop -d

# Batch mode for scripts (one refresh)
htop -b -n 1 > /tmp/htop_output.txt

atop basics

# Start atop (default sample every 60 s)
atop

# Switch views (g: process, m: memory, d: disk, n: network, c: CPU, j: JVM)
# Move forward/backward one sample: t / T
# Read historic data (default 30 days retained)
atop -r /var/log/atop/atop_20230511 -b 09:30 -e 10:30

Real‑world case – locate a CPU hog

# Scenario: CPU >90 %
htop
# Press F3 or '/' and type 'java' to filter
# If several java processes appear, press 't' for tree view and identify the child consuming CPU
# Select the offending process and press 'k', choose signal 15 (TERM) first, then 9 (KILL) if needed
# For deeper analysis:
strace -p <PID>

Tool 2: iotop – Interactive Disk I/O Monitoring

Scenario

Disk I/O is high but the responsible process is unknown. iostat shows high %util without per‑process attribution.

Installation

# RHEL/CentOS
yum install iotop -y

# Debian/Ubuntu
apt install iotop -y

Core Usage

Interactive mode

# Run iotop (requires root)
sudo iotop

# Shortcuts
# ↑↓   select process
# Space pause/resume
# r     reverse sort order
# o     show only processes doing I/O
# a     show accumulated I/O (no speed)
# q     quit

Non‑interactive mode for scripts

# Refresh every second
sudo iotop -b

# Refresh 5 times
sudo iotop -b -n 5

# Show only I/O‑active processes
sudo iotop -b -o

# Custom interval (0.5 s)
sudo iotop -b -d 0.5

# Filter by user
sudo iotop -b -u mysql

# Show accumulated I/O (overall consumption)
sudo iotop -b -a

# Monitor a specific PID continuously
watch -n 1 'sudo iotop -b -p 12345'

Case – locate the process causing heavy write I/O on a log server

# Identify I/O‑heavy process
sudo iotop -b -o -n 5
# Example output shows rsyslogd with 45 MB/s write
# Inspect rsyslog configuration
cat /etc/rsyslog.conf
# Check daily log growth
du -sh /var/log/*
# Mitigations (example):
# - Enable log compression
# - Adjust log‑rotation interval
# - Move logs to SSD

Tool 3: nethogs – Per‑Process Network Traffic

Scenario

Bandwidth is saturated, but iftop only shows IP‑level traffic, not the responsible process.

Installation

# RHEL/CentOS (EPEL required)
yum install nethogs -y

# Debian/Ubuntu
apt install nethogs -y

Core Usage

# Start nethogs (requires root)
sudo nethogs

# Monitor a specific interface
sudo nethogs eth0

# Monitor multiple interfaces
sudo nethogs eth0 eth1

# Shortcuts
# q   quit
# m   switch unit (KB/s, MB/s, B/s)
# r   sort by received traffic
# s   sort by sent traffic

Non‑interactive logging

# Capture for 60 s and write to file
sudo nethogs -d 1 -c 60 > /tmp/nethogs.log &

Case – backup script hogging bandwidth at night

# Run nethogs during backup window
sudo nethogs -d 1 > /tmp/nethogs_backup.log &
# Analyse log after backup
grep -v "Kbit" /tmp/nethogs_backup.log | awk '{print $2, $10}' | sort -rn | head -10
# Result shows rsync as top consumer
# Mitigation: limit rsync speed
rsync -avz --bwlimit=50000 /data/ backup-server:/backup/

Tool 4: ncdu – Disk Usage Analyzer

Scenario

The root partition is full and the offending directory is unknown. df shows overall usage, while du on each directory is too slow.

Installation

# RHEL/CentOS
yum install ncdu -y

# Debian/Ubuntu
apt install ncdu -y

Core Usage

# Scan the whole filesystem (may take a while)
sudo ncdu /

# Scan a specific directory
sudo ncdu /var

# Scan the current directory
ncdu

# Navigation shortcuts
# ↑↓   move cursor
# Enter go into sub‑directory
# ←    go back
# n     sort by name
# s     sort by size
# C     sort by item count
# d     delete selected file/dir (dangerous!)
# t     toggle display of sub‑directory total size
# g     show percentage bar
# q     quit

Advanced options

# Fast scan without detailed info (good for huge directories)
ncdu -1 /

# Show only items larger than 100 MB
ncdu --max-depth 1 /

# Exclude special filesystems (e.g., /proc, /sys)
ncdu -x /

# Export results for offline analysis
ncdu -o /tmp/ncdu.txt /
# Import and view saved results
ncdu -r /tmp/ncdu.txt

# Silent mode (no progress display)
ncdu -q /

# Combine with find to locate huge files
find / -type f -size +1G -exec ls -lh {} \;

Case – root partition full

# Verify overall usage
df -h /
# Analyse root with ncdu, sort by size
sudo ncdu -x /
# Drill down into large directories (e.g., /var/log)
# Verify before deletion
ls -lht /var/log/* | head
# View tail of a log to ensure safe removal
tail /var/log/messages

Tool 5: strace – System Call Tracing

Scenario

A program hangs after start or exits abruptly without logs. The slow part of the execution is unknown.

Installation

# RHEL/CentOS
yum install strace -y

# Debian/Ubuntu
apt install strace -y

Core Usage

# Trace a command execution
strace ls /tmp

# Attach to an existing process (known PID)
strace -p 12345

# Follow forked children
strace -f -p 12345

# Common options
# -c   summarize time, count, errors per syscall
# -f   follow forks
# -p PID   attach to PID
# -o FILE  write output to file
# -t   show timestamps (seconds)
# -tt  show timestamps (microseconds)
# -T   show time spent in each call
# -e trace=open,read,write   trace only selected syscalls
# -e trace=network           trace network‑related calls
# -e trace=file              trace file‑related calls
# -e trace=process           trace process‑related calls
# -e signal=ALL             trace all signals

Example output

# execve("/bin/ls", ["ls","/tmp"], ...) = 0
# open("/etc/ld.so.cache", O_RDONLY) = 3
# getdents(3, /*12 entries*/, 32768) = 368
# write(1, "file1
file2
", 12) = 12
# +++ exited with 0 +++

Practical scenarios

# Locate where a program hangs
strace -f -o /tmp/strace.log my_program
# After Ctrl+C, inspect the log
tail -100 /tmp/strace.log

# Identify slow syscalls
strace -c -T -p 12345

# Trace file accesses only
strace -e trace=open,openat,read,write -p 12345

# Trace network activity
strace -e trace=network -p 12345

# Trace all signals
strace -e signal=ALL -p 12345

# Find configuration files read by a process
strace -e trace=open,openat -p 12345 2>&1 | grep "\.conf"

Performance considerations

# strace significantly slows the target program; avoid in production
# High‑frequency calls generate massive output
# Long tracing sessions can create huge log files
# In containers, insufficient permissions may prevent tracing

Tool 6: lsof – List Open Files

Scenario

Attempting to delete a file fails with “file is busy”. Need to know which process holds a port or file.

Installation

# RHEL/CentOS
yum install lsof -y

# Debian/Ubuntu
apt install lsof -y

Core Usage

# List all open files (requires root for full view)
sudo lsof

# Files opened by a specific user
sudo lsof -u username

# Files opened by a specific PID
sudo lsof -p 12345

# Which process is listening on port 80
sudo lsof -i :80

# Which process is using a specific file
sudo lsof /var/log/messages

# List all network files
sudo lsof -i

# Show all network connections with numeric ports/hosts
sudo lsof -i -P -n

Advanced usage

# Find deleted files that are still open (common log‑file issue)
sudo lsof | grep deleted
# Example output shows mysqld holding a deleted slow.log
# Resolve by restarting the service or sending SIGKILL

# List open files on a specific filesystem
sudo lsof +f -- /data

# List open files on a specific device type
sudo lsof +D /dev/

# Show sockets opened by a process
sudo lsof -p 12345 -U

# Show file‑type statistics
sudo lsof -s

Case – "file is busy" error

# Attempt to delete
rm /tmp/test.log
# Error: cannot remove '/tmp/test.log': Text file busy
# Find the holder
sudo lsof /tmp/test.log
# Example output shows bash PID 12345 holding the file
# Resolve by terminating the process gracefully
kill -TERM 12345   # ask the process to close the file
# If needed, force kill
kill -9 12345

Case – log file not releasing space after deletion

# Log file deleted but disk space not freed
sudo lsof | grep deleted
# Example shows rsyslogd still holding the file
# Solutions:
#   - Restart rsyslog (may lose recent logs)
#   - Send HUP to rsyslog to reopen logs
systemctl restart rsyslog
# or
killall -HUP rsyslogd
# Verify no more deleted‑open files
sudo lsof | grep deleted | grep rsyslog

Tool 7: tcpdump – Network Packet Capture

Scenario

The server experiences packet loss, latency, or suspected attacks. Need to capture raw packets for analysis.

Installation

# RHEL/CentOS
yum install tcpdump -y

# Debian/Ubuntu
apt install tcpdump -y

Core Usage

# Capture all packets (requires root)
sudo tcpdump

# Capture on a specific interface
sudo tcpdump -i eth0

# Capture a limited number of packets
sudo tcpdump -i eth0 -c 100

# Save to a file for later analysis with Wireshark
sudo tcpdump -i eth0 -w /tmp/capture.pcap

# Read a capture file
sudo tcpdump -r /tmp/capture.pcap

# Capture only TCP/UDP/ICMP packets
sudo tcpdump -i eth0 tcp
sudo tcpdump -i eth0 udp
sudo tcpdump -i eth0 icmp

BPF filters (core feature)

# Capture a specific port
sudo tcpdump -i eth0 port 80

# Capture traffic to/from a specific host
sudo tcpdump -i eth0 host 192.168.1.100

# Capture only source IP
sudo tcpdump -i eth0 src host 192.168.1.100

# Capture only destination IP
sudo tcpdump -i eth0 dst host 192.168.1.100

# Combine conditions
sudo tcpdump -i eth0 host 192.168.1.100 and port 80
sudo tcpdump -i eth0 port 80 or port 443
sudo tcpdump -i eth0 src host 192.168.1.100 and not port 22

# Capture HTTP GET requests (hex pattern for "GE")
sudo tcpdump -i eth0 -A 'tcp[((tcp[12:1] & 0xf0) >> 2):2] = 0x4745'

# Capture HTTP Host header (hex for "HTTP")
sudo tcpdump -i eth0 -A 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450' | grep Host

Output format example

# 10:30:00.123456 IP 192.168.1.100.54321 > 192.168.1.1.80: Flags [S], seq 12345, win 65535, length 0

Fields: timestamp, protocol, source IP:port, direction arrow, destination IP:port, TCP flags, sequence number, window size, payload length.

Practical scenarios

# Diagnose slow web response
sudo tcpdump -i eth0 -A port 80 | grep -E "GET|200 OK"

# Check TCP three‑way handshake problems
sudo tcpdump -i eth0 'tcp[tcpflags] = tcp-syn' | grep retransmission

# Detect duplicate ACKs (loss indicator)
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-ack != 0' | head

# Measure connection establishment time
sudo tcpdump -i eth0 'tcp[tcpflags] = tcp-syn' -T statistics

# Capture ICMP (ping) traffic
sudo tcpdump -i eth0 icmp

# Capture DNS queries
sudo tcpdump -i eth0 port 53

Integration with Wireshark

# Capture on server, then download for analysis
sudo tcpdump -i eth0 -w /tmp/capture.pcap host 192.168.1.100 and port 80
scp user@server:/tmp/capture.pcap ./
wireshark capture.pcap

# Stream directly to Wireshark from a remote host
wireshark -k -i <(ssh user@server "tcpdump -i eth0 -w - host 192.168.1.100")

Tool 8: netstat / ss – Network Connection Inspection

Scenario

Need to see which connections are ESTABLISHED, how many are in TIME_WAIT, which ports are listening, and whether socket queues are backed up.

Installation

# Usually pre‑installed. If not:
yum install net-tools iproute -y   # provides netstat and ss
apt install net-tools iproute2 -y

Core Usage – netstat

# All connections
netstat -an

# Listening ports only
netstat -ln

# TCP connections
netstat -tn

# UDP connections
netstat -un

# Socket summary
netstat -s

# Routing table
netstat -r

# Interface statistics
netstat -i

# Show process info (requires root)
netstat -tnp

# Count connections per state
netstat -tn | awk '/^tcp/ {print $6}' | sort | uniq -c | sort -rn

Core Usage – ss (modern replacement)

# All connections (faster than netstat)
ss -an

# Listening ports
ss -ln

# TCP connections
ss -tn

# UDP connections
ss -un

# Socket summary
ss -s

# Show process info
ss -tnp

# Detailed timer information (retransmissions, timeouts)
ss -ti

# Memory usage per socket
ss -m

# Count connections per state
ss -tn | awk '/^tcp/ {print $1}' | sort | uniq -c | sort -rn

# Show connections for a specific port (e.g., 80)
ss -tn sport = :80 or dport = :80

# Show established connections on port 80
ss -tn state established '( sport = :80 or dport = :80 )'

Connection state analysis

# Count TIME_WAIT connections
ss -tn state time-wait | wc -l

# Count ESTABLISHED connections
ss -tn state established | wc -l

# Large TIME_WAIT often indicates many short‑lived connections; mitigation:
#   1. Enable TIME_WAIT reuse (net.ipv4.tcp_tw_reuse=1)
#   2. Use persistent connections on client side
#   3. Tune kernel parameters (net.ipv4.tcp_fin_timeout)

# View half‑open (SYN_RECV) connections – possible backlog issue
ss -tn state syn-recv

# View listen backlog (full‑connection queue)
ss -ltn sport = :80

Socket buffer analysis

# Show receive and send queue sizes plus memory usage
ss -tm
# Example line:
# ESTAB 0 0 192.168.1.1:80 192.168.1.100:54321 mem:(r4368000,w8736000)
# Recv‑Q / Send‑Q indicate data waiting in kernel queues
# mem:(r...,w...) shows buffer sizes in bytes
# Monitor memory usage over time
watch -n 1 'ss -s'

Case – excessive connections on a web server

# Overall TCP stats
ss -s
# Observe many TIME_WAIT entries
ss -tn state time-wait | awk '{print $4}' | cut -d: -f2 | sort | uniq -c | sort -rn | head
# Identify IPs with many ESTABLISHED connections
ss -tn state established | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head
# If a single IP dominates, block it
iptables -I INPUT -s 1.2.3.4 -j DROP
# Tune kernel parameters for TIME_WAIT reuse
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
# Increase listen backlog
echo 4096 > /proc/sys/net/core/somaxconn

Tool 9: curl – HTTP and Network Diagnostics

Scenario

Need to test an HTTP endpoint, view response headers, measure latency, verify DNS resolution, or simulate various request types.

Core Usage

# Simple GET
curl https://example.com

# Show response headers
curl -i https://example.com

# Show only headers
curl -I https://example.com

# Verbose (request + response headers)
curl -v https://example.com

# Follow redirects
curl -L https://example.com

# Save output to file
curl -o /tmp/output.html https://example.com

# Download with original filename
curl -O https://example.com/file.tar.gz

# Silent mode (no progress bar)
curl -s -O https://example.com/file.tar.gz

POST examples

# Form data
curl -X POST -d "username=admin&password=123456" https://example.com/login

# JSON payload
curl -X POST -H "Content-Type: application/json" \
    -d '{"username":"admin","password":"123456"}' \
    https://example.com/api/login

# Basic authentication
curl -X POST -u user:pass -d "data=value" https://example.com/api

# File upload
curl -X POST -F "file=@/tmp/test.txt" https://example.com/upload

# Custom headers
curl -H "Authorization: Bearer token123" -H "X-Custom-Header: value" https://example.com/api

Advanced options

# Limit download rate to 100 KB/s
curl --limit-rate 100k -O https://example.com/largefile.tar.gz

# Maximum total time (30 s)
curl -m 30 -O https://example.com/file

# Connection timeout (10 s)
curl --connect-timeout 10 -I https://example.com

# Use HTTP proxy
curl -x http://proxy.example.com:8080 https://example.com

# Send cookies
curl -b "session=abc123" https://example.com/api

# Save cookies to file
curl -c /tmp/cookies.txt https://example.com/login
# Use saved cookies
curl -b /tmp/cookies.txt https://example.com/api

# Skip TLS verification (testing only)
curl -k https://example.com
# Force TLS 1.2
curl --tlsv1.2 https://example.com
# Specify CA bundle
curl --cacert /etc/ssl/certs/ca-certificates.crt https://example.com

HTTP health‑check script (example)

#!/bin/bash
check_http() {
  local url=$1 name=${2:-$url} expected=${3:-200}
  response=$(curl -o /dev/null -s -w "%{http_code}|%{time_total}|%{size_download}" \
    --connect-timeout 5 -L "$url")
  code=$(echo $response | cut -d'|' -f1)
  time=$(echo $response | cut -d'|' -f2)
  size=$(echo $response | cut -d'|' -f3)
  if [ "$code" = "$expected" ]; then
    echo "[OK] $name – HTTP $code – ${time}s – ${size}bytes"
    return 0
  else
    echo "[FAIL] $name – Expected $expected, got $code – ${time}s"
    return 1
  fi
}

# Example checks
check_http "https://example.com" "Main Site" 200
check_http "https://example.com/api/health" "API Health" 200
check_http "https://example.com/static/test.js" "Static JS" 200
check_http "https://example.com/notexist" "404 Test" 404

DNS debugging with dig

# Query a specific DNS server
dig @8.8.8.8 example.com

# Show only answer section
dig example.com +short

# Show all records
dig example.com ANY

# MX records
dig example.com MX

# Reverse lookup
dig -x 93.184.216.34

# Batch check multiple domains
for domain in example.com google.com github.com; do
  ip=$(dig +short $domain)
  echo "$domain -> $ip"
done

CDN verification

# View response headers to identify CDN
curl -I https://example.com
# Common CDN headers:
#   Cloudflare – cf-ray, cf-cache-status
#   Akamai – x-akamai-transformed, server
#   CloudFront – x-cache, via
#   Alibaba CDN – x-swift-optimized, x-oss
# Compare direct IP vs CDN
origin_ip=$(dig +short example.com)
curl -H "Host: example.com" http://$origin_ip/

Tool 10: systemctl + journalctl – Service and Log Management

Scenario

A service fails to start or runs slowly; need to view historical logs, identify who triggered a restart, and inspect real‑time output.

Core Usage – systemctl

# Service status
systemctl status nginx

# Start / stop / restart
sudo systemctl start nginx
sudo systemctl stop nginx
sudo systemctl restart nginx

# Reload configuration without dropping connections
sudo systemctl reload nginx

# Check if active / enabled at boot
systemctl is-active nginx
systemctl is-enabled nginx

# Enable or disable at boot
sudo systemctl enable nginx
sudo systemctl disable nginx

# Reload systemd manager configuration
sudo systemctl daemon-reload

# List all running services
systemctl list-units --type=service --state=running

# List failed services
systemctl --failed --type=service

Core Usage – journalctl

# View all logs from the beginning
sudo journalctl

# View newest entries first
sudo journalctl -e

# Follow logs in real time
sudo journalctl -f

# Logs for a specific service
sudo journalctl -u nginx

# Error‑level logs for a service
sudo journalctl -u nginx -p err

# Time filtering
sudo journalctl --since "2026-05-11 10:00:00"
sudo journalctl --since "1 hour ago"
sudo journalctl --since today
sudo journalctl --since yesterday
sudo journalctl --since "2026-05-01" --until "2026-05-02"

# Logs from previous boot
sudo journalctl -b -1

# List boot IDs
sudo journalctl --list-boots
# Show logs for a specific boot ID
sudo journalctl -b abc123def456

# Disk usage of the journal
sudo journalctl --disk-usage
# Vacuum old logs (keep last 7 days)
sudo journalctl --vacuum-time=7d
# Vacuum to size limit (e.g., 500 M)
sudo journalctl --vacuum-size=500M
# Keep only last 5 journal files
sudo journalctl --vacuum-files=5

Combined troubleshooting example – nginx fails to start

# 1. Check service status
sudo systemctl status nginx
# 2. View recent error logs
sudo journalctl -u nginx -p err --no-pager -n 100
# 3. Test configuration syntax
sudo nginx -t
# 4. If port 80 is occupied, find the holder
sudo lsof -i :80
# 5. Kill the conflicting process
sudo kill $(sudo lsof -t -i :80)
# 6. Restart nginx and verify
sudo systemctl restart nginx
sudo systemctl status nginx
curl -I localhost

Service management advanced

# Show service dependencies
systemctl list-dependencies nginx
# Show reverse dependencies (who depends on this service)
systemctl list-dependencies --reverse nginx
# Show resource limits for a service
systemctl show nginx | grep -E "Memory|Limit"
# Temporarily adjust memory limit
sudo systemctl set-property nginx MemoryLimit=512M
# Show service start timestamp
systemctl show nginx -p ActiveEnterTimestamp
# View full unit properties (including defaults)
systemctl show nginx

Case – MySQL stops unexpectedly (OOM)

# Service status
sudo systemctl status mysql
# Find when and why it stopped
sudo journalctl -u mysql --since "1 day ago" | grep -E "Stopping|Started|Killed|OOM"
# Check kernel OOM messages
sudo journalctl -k | grep -i oom
dmesg | grep -i oom
# Inspect resource limits
systemctl show mysql | grep -E "Memory|Limit"
# If OOM, possible mitigations:
#   - Add RAM
#   - Reduce innodb_buffer_pool_size
#   - Add swap space
#   - Adjust OOM score

Quick Reference

htop

– interactive process monitor (e.g., htop -u mysql, htop -t) atop – comprehensive system monitor (e.g., atop -r /var/log/atop/xxx) iotop – per‑process I/O monitor (e.g., iotop -b -o, iotop -b -u mysql) nethogs – per‑process network traffic (e.g., nethogs eth0) ncdu – disk usage analysis (e.g., ncdu /, ncdu -x /) strace – system call tracing (e.g., strace -p PID, strace -c command) lsof – list open files (e.g., lsof -i :80, lsof -p PID) tcpdump – network packet capture (e.g., tcpdump -i eth0 port 80 -w file.pcap) ss – fast socket inspection (e.g., ss -tn, ss -s, ss -tnp) curl – HTTP diagnostics (e.g., curl -v, curl -I, curl -X POST) journalctl – systemd log viewer (e.g., journalctl -u svc -f, journalctl --since today)

Document Metadata

Version: 1.0

Update date: 2026‑05‑11

Supported systems: RHEL/CentOS 7‑9, Ubuntu 18.04/20.04/22.04

Target audience: Linux server daily operations