Essential Linux System Analysis Tools Every Engineer Should Master

Summary of Common Linux System Analysis Tools

For developers and system engineers, a Linux system can be examined through four critical modules: CPU, network, memory, and disk. This guide provides concise introductions to the most effective command‑line tools for each module, helping readers quickly understand their purpose, common options, and typical output.

1. CPU

1.1 top

The top command displays real‑time system performance metrics, including CPU usage, process list, and memory statistics. Frequently used options:

-c : show full command line

-d : set refresh interval

-i<seconds> : set delay time

-u<user> : filter by user

-p<pid> : monitor specific process

-n<count> : limit number of iterations

Typical top output is shown below:

1.2 strace

strace

traces system calls made by a process, reporting arguments, return values, and execution time. Important options include:

-c : summarize call counts and time

-f : follow child processes created by fork

-t / -tt : prepend timestamps (seconds / microseconds)

-T : display time spent in each call

-e trace=<set> : limit tracing to specific call sets (e.g., file, process, network)

-o <file> : write output to a file

Example:

strace -f -o loadconfigure-strace.txt -e execve ./loadconfigure

1.3 perf

perf

is a powerful performance analysis suite. Common subcommands:

perf top : live view of hottest functions

perf record : record events for later analysis

perf report : display recorded data, optionally with call graphs

Typical perf record -g -p <pid> output is illustrated below:

1.4 vmstat

vmstat

provides a comprehensive view of processes, memory, swap, I/O, and CPU activity. Key fields include:

procs – r (run queue length), b (blocked processes)

memory – swap, free, buff, cache

io – bi (blocks in), bo (blocks out)

system – in (interrupts), cs (context switches)

cpu – us (user), sy (system), wa (IO wait), id (idle)

2. Network

2.1 netstat

netstat

prints network connections, routing tables, and interface statistics. Frequently used options:

-a : show all sockets

-t : display TCP only

-u : display UDP only

-l : show listening services

-p : show owning program

-r : display routing table

-c : continuous display at intervals

Example of TCP state explanation and typical netstat -te output is shown below:

2.2 lsof

lsof

lists open files and the processes that opened them, which includes network sockets because everything in Linux is a file. Important columns:

COMMAND – process name

PID – process ID

USER – owner

FD – file descriptor

TYPE – file type

SIZE – file size

NAME – file name

Typical usage: lsof -p 1234 or

lsof -i :80

2.3 tcpdump

tcpdump

captures network packets for analysis. Common options:

-i <iface> : select interface

-nn : show numeric IP and ports

-w <file> : write raw packets to file

-c <count> : capture a limited number of packets

-A : print packet contents as ASCII

-X : show hex and ASCII

Example capturing traffic on port 10014:

tcpdump -i eth1 'port 10014' -xxx

3. Memory

3.1 valgrind

valgrind

(especially its memcheck tool) detects memory errors such as use‑after‑free, uninitialized reads, and leaks. Typical options:

--leak-check=full : detailed leak report

-q : quiet mode

-v : verbose output

--track-fds=yes : trace file descriptors

Sample command:

valgrind --leak-check=full /path/to/app --config=/path/to/config

4. Disk

4.1 iotop

iotop

monitors per‑process I/O usage in a top‑like interface, complementing tools such as iostat that report device‑level statistics.

Typical iotop display:

By mastering these utilities, engineers can efficiently locate performance bottlenecks, diagnose failures, and optimize Linux services.