How to Identify and Fix Linux Server Performance Bottlenecks

1. Common server performance bottlenecks

CPU

Network I/O

Disk I/O

2. Optimization methods

Improving CPU performance:

Concurrency: use multithreading or multiprocessing; upgrade to NPTL; keep thread/process count no greater than CPU cores.

Use locks cautiously; improve architecture to avoid locks when possible.

Minimize expensive string operations such as sprintf and snprintf, which consume CPU for lexical analysis.

Reduce system calls (e.g., time) to lower user‑kernel transition overhead.

Avoid unnecessary traversal operations.

Simplify implementation based on real requirements.

Optimize architecture; consider a dedicated thread for costly string protocol parsing.

A good architecture keeps overall CPU consumption evenly distributed across cores, typically around 70% utilization.

Improving network I/O:

Replace select with epoll.

Develop using non‑blocking mode.

Improving disk I/O:

Leverage free memory as filesystem cache; larger memory improves storage performance.

Use sequential writes to reduce seek operations.

Apply cache strategies to fully utilize CPU and memory resources, alleviating disk read/write pressure.

3. Tools for locating bottlenecks

top – displays running processes, CPU usage, system load, memory usage, and can sort by CPU consumption; also shows per‑CPU distribution.

free – shows physical and swap memory usage, as well as buffers and cache.

vmstat – a comprehensive performance analysis tool that reports process status, memory usage, virtual memory, disk I/O, interrupts, context switches, and CPU usage.

Key vmstat fields:

Procs – r : number of running or runnable processes (high values may indicate need for more CPU); b : number of blocked processes, often due to I/O.

Memory – similar information to free .

Swap – si (swap‑in) and so (swap‑out); non‑zero values over time signal memory pressure that also impacts CPU and disk.

IO – bi (blocks read per second) and bo (blocks written per second); high values can increase CPU wait time during random disk access.

System – in (interrupts per second) and cs (context switches per second); higher values increase kernel CPU consumption.

CPU – us (user‑mode CPU %), sy (kernel‑mode CPU %), wa (I/O wait %), id (idle %). High us indicates heavy user‑process load; high sy points to kernel overhead; high wa reveals serious I/O waiting.

Strace – traces system calls and signals of a running process.

tcpdump – Linux packet capture tool; can redirect captured data to a file for later analysis with Wireshark (formerly Ethereal) on Windows.

gprof – profiles program functions, showing CPU time per function, call counts, and optionally a simple call graph.

Using gprof:

Compile with the -pg flag (e.g., gcc -pg program.c -o program).

Run the program to generate gmon.out.

Execute gprof -b program gmon.out | less to view profiling results.

Source: https://blog.csdn.net/michael_kong_nju/article/details/44880939