How to Diagnose Linux Performance Issues: Tools & Step‑by‑Step Guide

Analysis Tools

Linux provides many performance analysis utilities. Common tools include top, vmstat, iostat, strace, oprofile, slabtop, iptraf and others.

Analysis Methodology

Problem diagnosis in backend development usually starts with a high‑level view of CPU, memory, disk and network. A decision‑tree approach narrows the scope: begin from the observed symptom (e.g., high CPU) and follow the corresponding checklist.

CPU Issues

Kernel dominates CPU? Check kernel CPU share with top.

Frequent interrupts? Inspect /proc/interrupts or use procinfo.

Kernel hotspots? Use oprofile to locate heavy kernel functions.

User space dominates CPU? Verify with top.

Which process consumes most CPU? Sort processes in top.

Kernel vs. user time for a process? Run time and examine user/system percentages.

System calls taking time? Trace with strace or oprofile.

Functions taking time? Use ltrace or oprofile to identify hot functions and possible cache misses.

Memory Issues

Kernel memory growth? Inspect with slabtop.

Kernel memory type? Sort slabtop output to find large objects.

Process memory growth? Use top or ps and examine RSS.

Process memory composition? Read /proc/[pid]/status fields such as VmExe, VmLib, VmData.

Functions using large stack? Attach with gdb and compare stack pointers.

Functions allocating large heap? Use memprof to locate heap growth.

Large shared libraries? Examine /proc/[pid]/maps for library sizes.

Large text sections? Run nm to list symbol sizes.

Shared memory growth? Query with ipcs.

Processes using shared memory? Use ipcs -p to map owners.

Disk I/O Issues

Which process generates I/O? Find with iotop.

Which files are accessed? Trace the process with strace to see read/write syscalls and map file descriptors to paths.

Network I/O Issues

Network errors? Check interface statistics with ifconfig or ip.

Traffic type? Use iptraf to view protocol/port distribution.

Process handling traffic? Identify with netstat.

Remote source of traffic? Capture packets with etherape or wireshark.

Socket responsible? After locating the process, use strace or lsof to find the socket.