Why %CPU Is Misleading: Understanding the Real Meaning of CPU Utilization

Introduction: This article is a translation of Brendan Gregg's 2017 blog post "CPU Utilization is Wrong", which challenges the common interpretation of the %CPU metric.

1. Introduction

Many assume that a 90% CPU utilization means the processor is busy 90% of the time.

In reality, a large portion of that time may be spent stalled, waiting for resources such as memory.

Stalled cycles cannot execute instructions, so the program makes no progress despite a high %CPU reading.

2. What Does CPU Utilization Really Mean?

CPU utilization traditionally measures "non‑idle time", i.e., the time the CPU is not running the idle thread. The kernel records this during context switches.

If a non‑idle thread runs for 100 ms, the kernel may report 100% utilization for that interval, a metric inherited from time‑sharing systems.

Modern CPUs run far faster than memory, so memory‑access latency dominates the non‑idle time. A high %CPU often masks a memory bottleneck rather than a CPU bottleneck.

Historically, CPU frequency grew faster than DRAM latency, but around 2005 manufacturers shifted to multicore and multithreading, increasing memory demand and making stalls more common.

3. How to Truly Know What the CPU Is Doing?

Performance Monitoring Counters (PMCs) expose detailed CPU state. The figure below shows a 10‑second perf capture of all CPU states.

The key metric is IPC (instructions per cycle). An IPC of 0.78 on a processor capable of 4.0 IPC indicates the machine is only using about 19.5% of its theoretical capacity.

PMU events can help pinpoint the causes of stalls; understanding the micro‑architecture is essential.

4. Best Practices

If IPC < 1.0, the workload is likely memory‑bound. Reduce unnecessary memory accesses, improve cache hit rates, and prefer local memory.

If IPC > 1.0, the workload is compute‑bound. Reduce instruction count, eliminate redundant work, and consider flame‑graphs for analysis. Hardware options include higher clock rates, more cores, or hyper‑threading.

5. What Performance Tools Should Show

Tools that display %CPU should also show IPC, or split %CPU into instruction‑consuming cycles (%INS) and stalled cycles (%STL). Linux’s

tiptop

can display per‑CPU IPC.

6. Other Factors That Can Mislead CPU Utilization

Thermal throttling causing stalls.

Turbo Boost altering clock rates.

Kernel scheduling increasing clock speed.

Averaging effects: a minute‑average of 80% can hide short spikes of 100%.

Spin locks: high IPC but no real progress.

7. Update: Is CPU Utilization Really Wrong?

Discussion in the original blog’s comments and on sites like Hacker News and Reddit confirms that %CPU alone does not reveal the true bottleneck; it often hides memory stalls.

The metric is technically correct at the OS level but is easily misinterpreted. With hyper‑threading, stalled cycles may be used by another thread, further confusing %CPU readings.

8. Conclusion

%CPU has become a misleading metric because it includes cycles spent waiting for memory. Renaming it to %CYC (cycles) could reduce confusion.

Understanding %CPU requires complementary metrics such as IPC. IPC < 1.0 usually indicates a memory‑bound workload, while IPC > 1.0 suggests compute‑bound work. Performance tools should present both %CPU and IPC (or equivalent) to guide proper optimization.

Source: 内核月谈 Original article: http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html