Investigation of Slab Lock Issues and CPU Power Management in Linux Kernels for Storage Performance

Background: In Q4 2018 a batch of servers used for storage showed write latencies exceeding 800 ms (normally <300 ms) and sub‑optimal read performance.

Analysis revealed that the monitoring agent’s use of the ss command reads /proc/slabinfo , acquiring a large global slab lock; this blocks RocksDB writes and causes long lock waits. Two remedies exist: modify the monitoring agent or upgrade the kernel from 2.6 to 3.10/4.18, the latter being applied.

Read latency was linked to BIOS CPU performance‑mode settings; after tuning the BIOS (disabling power‑saving modes), I/O performance improved noticeably.

Slab Lock : Kernel 2.6.32 obtains a single coarse‑grained lock when reading slab information, leading to long contention. Kernels 3.10 and 4.18 split this into multiple fine‑grained locks, dramatically reducing lock time and mitigating impact on other processes.

CPU Power Management: Subsequent tests with the upgraded 4.18 kernel showed that the Q4 servers still performed poorly in read benchmarks. A small program measuring pread throughput indicated the 4.18‑kernel Q4 machine was the worst, while the 2.6‑kernel Q4 and older machines performed similarly.

A perf test script was used to evaluate raw CPU compute capability:

echo "i=0; while (( i < 1000000 )); do (( i ++ )); done;" | perf stat sh

Results showed that the Q4 machines were configured in BIOS energy‑saving mode, enabling P‑State, C‑State, and MONITOR/MWAIT settings. After disabling these power‑saving options, the 4.18‑kernel machine’s compute performance increased significantly, whereas the 2.6‑kernel machine’s performance degraded.

CPU power‑state categories:

P‑States (Performance States) adjust the core frequency and voltage; higher states (P0) deliver maximum performance, while lower states save power with minimal impact on latency‑tolerant workloads.

C‑States (CPU Sleep States) shut down parts of the core when idle; deeper states (C1, C3, C6) increase wake‑up latency and can affect latency‑sensitive services.

T‑States (Throttling States) trigger when the CPU exceeds safe temperature, intermittently halting instruction execution to cool down.

BIOS settings such as MONITOR/MWAIT allow deeper idle states with fewer interrupts compared to the traditional hlt instruction, but their impact on performance varies across workloads.

Further discussion on how different Linux kernels utilize these instructions and power‑states will be continued in the next article.