Why the Lsof Command Is an Underrated Lifesaver in Production

1. What Is lsof

lsof stands for "list open files"; in Linux everything is a file, so lsof can inspect network connections, devices, pipes, and regular files, making it a powerful system‑diagnostic tool.

2. Basic Usage

The simplest invocation lsof lists every open file, which can be overwhelming. Each output line contains fields such as COMMAND, PID, USER, FD, TYPE, DEVICE, SIZE/OFF, NODE, and NAME.

3. Network‑Related Tricks

Common commands:

# Show which process uses port 80
lsof -i:80
# List all TCP connections
lsof -i tcp
# List all UDP connections
lsof -i udp
# Query a specific IP and port
lsof [email protected]:22

Example: a Redis instance reported port 6379 as occupied; lsof -i:6379 revealed a zombie process, which was killed to resolve the issue.

4. Process‑Related Usage

Inspect files opened by a process:

# Files opened by PID 1234
lsof -p 1234
# Files opened by processes named nginx
lsof -c nginx
# Files opened by user www-data
lsof -u www-data

Case: a Java application suffered a memory leak; lsof -p showed a growing number of temporary files, pinpointing the leak to unclosed file streams.

5. Filesystem‑Related Tricks

Find deleted files that are still held open:

# Simple grep
lsof | grep deleted
# More precise
lsof +L1

Scenario: disk usage spiked despite deleting large logs; lsof | grep deleted identified a process still writing to a removed log file, and restarting the process freed the space.

6. Real‑World Cases

Case 1 – Port Conflict : Deploying a new service on port 8080 failed; lsof -i:8080 showed Jenkins occupying the port via an extra plugin.

Case 2 – File‑Handle Leak : A Python app reported "Too many open files"; after locating the PID, lsof -p 12345 | wc -l revealed a rising count of temporary files, traced to missing cleanup code.

Case 3 – Disk‑Space Anomaly : Sudden 95% disk usage;

lsof | awk '$7 ~ /^[0-9]+$/ && $7 > 1000000 {print $2, $7, $9}' | sort -k2 -nr

uncovered a deleted but still‑written‑to multi‑GB file.

7. Advanced Usage and Tips

Combine conditions (default OR, use -a for AND):

# OR relationship
lsof -u www-data -i
# AND relationship
lsof -a -u www-data -i

Control output format:

# Show IPs only
lsof -n -i
# Show port numbers only
lsof -P -i
# Combine
lsof -nP -i:80

Continuous monitoring:

# Refresh every 2 seconds
lsof -r 2 -i:80
# Exit when no output
lsof -r 1 -i:80

8. Performance Optimisation

Use specific parameters to avoid full scans.

Apply -n and -P to skip DNS and service‑name resolution.

Cache results when scripting.

# Faster
lsof -nP -i:80
# Slower
lsof | grep :80

9. Common Pitfalls

Permission: Some information requires root.

System Load: Running lsof on heavily loaded systems can affect performance.

Output Interpretation: Understanding fields, especially FD, is essential.

10. FD Field Details

cwd: Current working directory.

txt: Program code.

mem: Memory‑mapped file.

Numbers: File descriptor numbers.

r, w, u: Read, write, read/write modes.

11. Combining with Other Tools

# With ps
lsof -p $(pgrep nginx)
# Compare with netstat
netstat -tlnp | grep :80
lsof -i:80
# Process with awk
lsof -i | awk '/ESTABLISHED/ {print $2}' | sort | uniq -c

Using lsof together with scripts greatly speeds up troubleshooting.

12. Conclusion

lsof is an often‑underestimated utility that can resolve most file and network issues—from port conflicts to handle leaks and disk‑space mysteries. While it reveals symptoms, deeper analysis of system behavior remains essential, and practicing with realistic scenarios sharpens one’s troubleshooting skills.