Master Linux System Performance: Essential Tools, Commands, and Real‑World Analysis

Linux troubleshooting toolbox

A compact set of command‑line utilities can cover most interview questions about diagnosing Linux system problems. The tools are grouped by the resource they observe – CPU, memory, disk, network and processes – and each entry includes installation notes (where applicable) and a short usage example.

CPU monitoring

top : Real‑time view of CPU, memory and per‑process resource consumption. top htop (install with sudo apt install htop on Debian/Ubuntu): Enhanced UI with mouse support and tree view of threads. htop mpstat (part of sysstat): Per‑core utilisation. Example shows all cores refreshed every second. mpstat -P ALL 1 perf : Kernel‑level performance profiling (hotspots, cache misses).

perf top          # live hotspot view
perf record -g   # record a trace for later flame‑graph

sar (sysstat): Long‑term activity reporting. Example records CPU utilisation for five intervals.

sar -u 1 5

Memory monitoring

free : Quick snapshot of RAM and swap usage. free -h vmstat : Combined view of memory, swap, process queues and I/O. Useful for spotting CPU‑bound queues (r) and I/O wait (wa). vmstat 1 smem (install with sudo apt install smem): Shows per‑process USS/PSS/RSS values. smem -u slabtop : Inspects kernel slab cache usage.

slabtop -s c

Disk monitoring

iostat (sysstat): Throughput, latency and utilisation per block device. iostat -xz 1 iotop (install with sudo apt install iotop): Real‑time per‑process I/O activity. iotop -o blktrace : Low‑level block‑device I/O tracing for deep performance analysis.

blktrace -d /dev/sda -o trace

Network monitoring

ifconfig / ip : Show and configure network interfaces.

netstat : Legacy connection, routing and protocol statistics.

ss : Faster replacement for netstat. ss -tunlp iftop (install with sudo apt install iftop): Bandwidth per connection. sudo iftop -i eth0 nethogs (install with sudo apt install nethogs): Per‑process network traffic. nethogs eth0 tcpdump : Packet capture for deep analysis (requires root).

tcpdump -i eth0 port 80

Process inspection

ps and pstree : List processes and show hierarchy.

kill : Terminate a process.

strace : Trace system calls and signals of a running process. strace -p 1234 -T pidstat (sysstat): Per‑process CPU, memory and I/O statistics.

pidstat -d -p 1234 1

Typical troubleshooting flow

When a system feels slow, start with a high‑level view ( top or htop) to see overall load. If CPU appears saturated, drill down with mpstat or perf. If memory pressure is suspected, examine free and vmstat. Disk‑related slowness is investigated with iostat and iotop. Network latency problems start with iftop or nethogs, then use tcpdump for packet‑level details. For a stuck process, combine strace -p <PID> with perf record -g -p <PID> to see system calls and CPU hotspots.

Real‑world example: vmstat analysis

The vmstat output contains the following key fields:

r : Number of processes waiting for CPU. If r > number of CPU cores, the CPU is a bottleneck.

b : Processes in uninterruptible sleep (usually I/O wait).

swpd : Amount of swap used (KB). Non‑zero indicates swapping.

free : Free memory (KB).

si / so : Swap‑in and swap‑out rates (KB/s).

bi / bo : Blocks received from / sent to block devices (blocks/s).

us , sy , id , wa , st : CPU time spent in user mode, system mode, idle, I/O wait and stolen (virtualised) time.

Interpretation example:

If r is 0 and id is ~98 %, the system is idle.

High wa (>5 %) signals I/O contention.

Non‑zero si / so means the system is swapping, which degrades performance.

Real‑world example: iostat analysis

The iostat -d 3 output shows per‑device statistics:

Device : Name of the block device (e.g., vda, vdb).

tps : Transactions per second – total I/O operations per second.

kB_read/s and kB_wrtn/s : Read and write throughput.

kB_dscd/s : Discarded data (TRIM) – usually 0 on HDD/SSD unless the filesystem issues discard commands.

kB_read and kB_wrtn : Cumulative data read/written since boot.

Typical interpretation:

A high tps or kB_wrtn/s on a device indicates a write‑intensive workload; consider checking the application’s write pattern or the storage tier.

Low activity on a device ( tps ≈ 0) means it is not a performance bottleneck.

Combined monitoring tools

glances (install with sudo apt install glances): One‑stop, curses‑based dashboard that shows CPU, memory, disk I/O, network and process metrics.

dstat (install with sudo apt install dstat): Real‑time mix of vmstat, iostat and netstat.

dstat -c -m -d -n

Practical command snippets

Top‑10 CPU‑hungry processes: ps aux --sort=-%cpu | head -10 Top‑10 memory‑hungry processes: ps aux --sort=-%mem | head -10 Continuously watch disk usage:

watch -n 1 'df -h'

Conclusion

This toolbox equips you to answer interview questions and to diagnose real‑world Linux performance issues. By starting with a high‑level view and progressively drilling down using the commands above, you can quickly isolate CPU, memory, I/O or network bottlenecks and obtain the detailed information needed for remediation.