11 Linux Commands That Save You in Production: Disk, Logs, Ports & Ghost Processes

1. find — Precisely Locate Large Files and Abnormal Logs (Core for Disk Full)

Scenario: Disk at 100%, log explosion, service write failures, missing junk files.

Full Command:

# Full disk scan for files larger than 1G
find / -type f -size +1G -exec ls -lh {} \;

Field Breakdown: find: File search command. /: Start from root directory for full disk scan (widest scope, suitable for full-disk explosion investigation). -type f: Only find files, ignore directories, avoid interference. -size +1G: Filter files larger than 1G, precisely lock giant logs/junk files. -exec ls -lh {} \;: After finding, automatically display file size, permissions, time — no second query needed.

Other High-Frequency Variants:

# Search specific log directory for files larger than 500M
find /home/logs -type f -size +500M -exec ls -lh {} \;

/home/logs

: Limit search directory, avoid full disk scan, faster troubleshooting.

# Find files added/modified in last 3 days
find /home/project -mtime -3 -type f

-mtime -3

: Only show files changed within 3 days, targets sudden disk explosion and temporary junk files.

On-Site Summary: df checks disk, du checks directory, find checks file — three-step closed loop for all disk explosion issues.

Taboo: Never directly delete logs that are actively being written to; this causes disk space not to be released.

2. du -sh * — Step-by-Step Investigation of Large Directories

Scenario: Disk full, quickly locate which directory consumes space, narrow find scope.

Full Command: du -sh * | sort -hr Field Breakdown: du: Estimate file/directory disk usage. -s: Summarize, do not show sub-file details, clean output. -h: Human-readable units (G/M/K), no manual byte conversion. *: Stat all files and folders in current directory. sort: Sort. -hr: Sort by capacity descending, largest directory at a glance.

Practical Logic: First lock down large directory with this command, then cd into it and use find to pinpoint specific files — troubleshooting efficiency doubles.

3. tail -f — Real-Time Log Monitoring (First Entry for Troubleshooting)

Scenario: Reproduce errors, catch real-time exceptions, quickly locate API errors.

High-Frequency Command: tail -n 500 -f app.log Field Breakdown: tail: View file end (logs always append at end). -n 500: Load only last 500 lines, avoid terminal freeze from huge logs. -f: Real-time follow, new logs auto-appear. app.log: Specify log file name.

Error-Filtering Upgrade:

tail -n 500 -f app.log | grep -E "Exception|超时|Duplicate"

|

: Pipe, pass previous command output to next. grep: Keyword filter. -E: Support multiple keywords, monitor multiple error types simultaneously.

On-Site Iron Rule: All production issues must start with real-time logs before reproduction — eliminate blind guessing and blind restarts.

4. grep — Search Historical Errors (Essential for Forensics and Postmortems)

Scenario: Intermittent errors, historical faults, sporadic bugs — real-time capture missed, search logs for evidence.

Core Command: grep -C 20 "Duplicate entry" app.log Field Breakdown: grep: Text keyword search. -C 20: Core life-saving parameter! Show error line plus 20 lines of context before and after. "Duplicate entry": Error keyword to search (replace with null pointer, timeout, etc.). app.log: Target log file.

Practical Value: A single error line lacks persuasiveness; context reveals input parameters and operation scenarios — precise blame-shifting without taking the blame.

5. ps -ef | grep — Verify Process Status

Scenario: Service starts with no response, fake startup, multi-process conflicts, service mysteriously offline.

Command: ps -ef | grep java Field Breakdown: ps: View system processes. -e: Show all processes. -f: Full format (PID, parent PID, startup command, user). grep java: Filter Java processes, precisely locate project services.

On-Site Judgment Logic: No process = startup failure; multiple processes = duplicate startup; process exists but unreachable = service hung.

6. ss / netstat — Port Investigation (Compatible with All Servers, Including ss-less Systems)

Scenario: Connection refused, port unreachable, service cannot connect to middleware, port occupancy investigation — compatible with old and new Linux servers.

Key Note: Some old CentOS 6, minimal installs, stripped-down systems lack ss command . Dedicated compatibility solution and installation guide provided to avoid "command not found" errors.

Commands:

# [New Server Preferred] ss command (efficient, fast)
ss -an | grep 8080

# [Old Server / No ss Mandatory] netstat compatible replacement (100% compatible all systems)
netstat -an | grep 8080

# One-click install ss if missing (temporary install, works immediately)
yum install iproute -y

Field Breakdown (Common to Both): ss / netstat: Two major Linux network port query tools; ss newer and faster, netstat full-version compatible. -a: Show all connections and listening ports, no missed states. -n: Numeric port display, no domain/host resolution, query speed doubles, no lag. grep 8080: Precisely filter target business/middleware port.

Practical Conclusion (Universal): No listening port = service not started/startup error; port listening = network, firewall, configuration issue.

On-Site Pitfall Avoidance: If "ss: command not found", abandon ss immediately, use netstat instead — no time wasted installing, on-site troubleshooting prioritizes usability.

7. kill -15 / kill -9 — Process Termination Commands

Scenario: Process hung, port occupied, restart fails, fake-dead service cleanup.

Commands:

kill -15 PID
kill -9 PID

Field Breakdown: kill: Terminate process command. -15: Graceful termination (default), notifies program to finish transactions, save data, release ports — production preferred . -9: Force kill, violent termination, no cleanup — only when hung . PID: Corresponding process ID (obtained via ps command).

On-Site Pitfall Avoidance: Newbies must not start with -9; high risk of data loss, file corruption, transaction rollback failure.

8. top — View Server Load

Scenario: API latency, batch timeouts, slow page loads — distinguish server issue from code issue.

Command + Breakdown: top Explanation: top is a dynamic system monitoring panel; no complex parameters needed. Focus on three core points:

CPU saturated → code infinite loop, slow SQL, high concurrency.

Memory saturated → memory leak, cache buildup.

Load normal but service stuck → pure code bug, unrelated to server.

9. dmesg -T — Investigate Mysterious Process Disappearance

Scenario: Service suddenly crashes, no error in logs, auto crash, restart fixes it.

Command: dmesg -T | grep -i kill Field Breakdown: dmesg: View kernel logs (kernel-level issues invisible in business logs are all here). -T: Convert timestamps to human-readable time, convenient for correlating with fault time. grep -i kill: Fuzzy search for kernel kill records; -i ignores case.

Practical Divine Logic: 90% of mysterious service disappearances are OOM (Out of Memory) — system actively kills process; business logs record nothing, only this command reveals it.

10. curl — Self-Test API to Divide Responsibility

Scenario: Frontend-backend blame game, external network unreachable, API errors — quickly define problem boundaries.

Command: curl http://127.0.0.1:8080/xxx/query Breakdown: curl: Initiate request from server locally, no dependency on browser, frontend, or external network. 127.0.0.1: Local loopback address, bypasses external network and firewall.

Blame-Shifting Iron Rule:

Local works but frontend fails → frontend/domain/external network issue; local fails → backend service/configuration issue.

11. lsof | grep deleted — Investigate Disk False-Full Faults (Production Hidden Pit King)

Scenario: df shows disk 100% full, du and find find no large files, disk space mysteriously occupied, logs deleted but space not released, service sluggish/hung.

Newbie Fatal Misconception: Disk full → delete logs → space does not decrease at all → situation worsens → eventually blamed for human error.

Full Practical Command:

# Query deleted but unreleased zombie files (exclusive for disk false-full)
lsof | grep deleted

Field Breakdown: lsof: List open files — query all files currently held by processes. |: Pipe, pass query results. grep deleted: Precisely filter deleted but still held by process, space not released zombie files.

Production Core Principle (Must Understand): Linux mechanism: when a file is held by a running process , even if you rm delete it, disk space is not released immediately . The file entry disappears, but the process still holds the file handle, causing disk false-full — no large files found yet space exhausted.

This is the most frequent, most hidden disk fault in production; du and find cannot detect it — only lsof can.

On-Site Landing Solutions:

# Solution 1: Safely empty zombie log file (production preferred, no service restart)
echo "" > occupied_log_filename

# Solution 2: During maintenance window, restart corresponding process/service to fully release space
kill -15 PID_holding_file

On-Site Pitfall Avoidance Iron Rule: Disk explosion priority: first check deleted zombie files, then delete logs ! Strictly forbidden to directly rm delete business logs or running logs that are being written — 90% of disk false-full accidents are caused by newbies recklessly deleting files.

Frontline Implementation Exclusive: Command + Principle Minimalist Mnemonics (Screenshot Recommended)

find : Coarse directory check with du, precise file search with find, +size filters large files.

du : -sh for summary, sort for ordering, quickly lock explosion directory.

tail : -f real-time follow, -n limit lines, filter keywords to catch errors.

grep : -C with context, investigate errors without missing evidence.

ps : -ef full process view, investigate fake startup, multi-process conflicts.

ss/netstat : -an check port listening, new/old server dual compatible, cure connection refused.

kill : 15 graceful stop service, 9 force kill hung process.

top : Check load to assign responsibility, distinguish environment vs code issues.

dmesg : Check kernel logs, cure mysterious service disappearance.

curl : Local self-test API, end frontend-backend blame game.

lsof deleted : Cure disk false-full, delete file but space not released hidden faults.

Final Words from Lao Tang

Why are others faster at troubleshooting and avoid blame? Not luck — they understand command principles, not rote memorization .

Only knowing copy-paste fails when parameters or scenarios change; understanding every parameter meaning lets you flexibly adapt commands on-site for precise root-cause location.

These 11 commands + parameter breakdowns, combined with the five major production error types, already cover 99% of on-site firefighting scenarios for implementation roles . Master this set, and you upgrade from "newbie who only knows restart" to "veteran who identifies root cause and assigns responsibility in three minutes".