Mastering Linux Performance: A Deep Dive into the top Command and Thread Analysis

When a service suddenly spikes in CPU or memory usage, the first step is to log into the server and identify the offending process. The article starts with a simulated incident where top reveals PID 2816 consuming excessive CPU, and further inspection with top -Hp 2816 shows thread 2825 also using high CPU.

Because thread IDs in Linux are displayed in decimal, the author demonstrates converting them to hexadecimal using Python, which is necessary when analyzing thread dump (DUMP) files that reference threads by their hex NID.

Multiple jstack dumps of the same PID are recommended, as thread states can change over time. By comparing dumps, one can see a thread holding a lock and another waiting for it, guiding developers to the code section where the lock is not released.

The article then provides a detailed breakdown of the top interface:

First line : system time vs. uptime; focus on uptime because frequent reboots can mask issues.

Second line : number of tasks, with special attention to zombie processes.

Third line : CPU usage summary.

Fourth/Fifth lines : memory information, distinguishing between buffer (data awaiting processing) and cache (cached results, e.g., from a database).

SWAP : indicates disk‑based memory extension; heavy swapping signals insufficient RAM.

Key columns in the process list are explained: PID, USER, PR, VIRT, RES, SHR, etc. Notably, RES shows the actual physical memory used by a process, and the true memory footprint is RES‑SHR.

Additional top metrics are clarified:

US/SY : user‑space vs. system‑space CPU usage.

NI : proportion of processes with adjusted nice values.

ID : idle time; WA indicates time waiting for I/O resources.

HI/SI : hardware and software interrupt percentages.

ST : steal time for virtual machines.

By mastering these details, engineers can efficiently diagnose performance anomalies, differentiate between genuine load spikes and false alarms, and take targeted actions to optimize service stability.