Linux Performance Analysis Tools and Troubleshooting Methods for Backend Development

Linux performance analysis is a routine task for backend developers, yet it often suffers from unstable reproducibility, difficulty in pinpointing issues, and long investigation times. This article compiles common Linux analysis tools and a systematic troubleshooting workflow covering CPU, memory, disk, and network aspects, presented as a mind‑map for easy reference.

Analysis Tools

Linux offers many performance analysis utilities. Some tools overlap in functionality, while others complement each other for specific metrics. The accompanying mind‑map (available via the keyword 0801 ) summarizes frequently used tools.

Analysis Methods

The troubleshooting process is not a strict linear path from CPU → Memory → Disk → Network. Instead, once the problematic domain is identified, the investigation proceeds directly within that area, following the steps illustrated in the mind‑map.

01 CPU Usage Issues

Is the kernel consuming most CPU? Use top to check the kernel CPU share.

Are many interrupts being handled? Inspect procinfo or cat /proc/interrupts for interrupt counts and the responsible devices.

Where is kernel time spent? Run oprofile to identify kernel functions that consume significant time, then map them to subsystems (memory, network, disk, etc.).

Is user space dominating CPU? Check top for high user‑space CPU usage.

Which process uses most CPU? Sort processes in top to find the top CPU consumer.

Kernel vs. user time for the process? Use time to see the split; >25% kernel time suggests kernel‑side investigation.

Which system calls consume the most time? Employ strace or oprofile to locate costly system calls.

Which functions within the process are expensive? Use ltrace or oprofile to pinpoint hot functions and assess cache behavior with cachegrind .

02 Memory Usage Issues

Is kernel memory growing? Run slabtop to monitor kernel memory consumption.

What type of kernel memory is increasing? Sort slabtop output to identify large objects (e.g., inode_cache ) and investigate their purpose.

Is a process’s memory increasing? Use top or ps to sort by RSS and observe trends.

What memory categories does the process use? Examine /proc/<pid>/status (VmExe, VmLib, VmData, etc.) to determine large executable, libraries, or heap usage.

Which functions use large stack space? Attach with gdb and compare stack pointers across calls.

Which functions allocate a lot of heap? Use memprof to find heap‑heavy allocations and detect leaks.

Which shared libraries are oversized? Inspect /proc/<pid>/maps to list loaded libraries and their sizes.

Which functions have large text sections? Run nm on the binary to sort symbols by size.

Is shared memory growing? Check with ipcs for large or increasing shared memory segments.

Which process owns the shared memory? Use ipcs -p to identify creator and users, then verify proper allocation and cleanup.

03 Disk I/O Usage Issues

Which process is performing disk I/O? Use iotop to locate high‑IO processes.

What files are being accessed? Trace the process with strace to capture file‑related syscalls and map file descriptors back to actual files.

04 Network I/O Usage Issues

Are network interfaces reporting many errors? Check with ifconfig or ip for error counters.

What type of traffic is flowing? Use iptraf to view protocol and port distribution.

Which process handles the traffic? Identify with netstat or ss the process bound to the relevant ports.

Is the traffic coming from a remote system? If no local process is responsible, consider external sources and investigate with etherape or wireshark .

Which socket is responsible? After locating the process, use strace or lsof to find the specific socket.

The article also invites readers to share their own Linux troubleshooting tips, offering a small prize for selected comments, and promotes the Tencent Cloud developer community for further resources.