Why Is Disk IOWait Spiking on Storm Nodes? Uncover kjournald, pdflush and Tuning Tips

Recently in production, real‑time Storm machines have shown intermittent spikes in disk IOWait. Storm processes data in memory, so disk I/O is limited to topology logs, and no other applications run on the nodes, prompting investigation into which processes are consuming disk I/O.

1. iotop

Running iotop reveals that the kjournald process consumes the most I/O resources, with several Java processes also contributing.

2. Inspecting /proc

cut -d " " -f 1,2,42/*/proc | sort -n -k +3

Sample output shows high I/O usage by kjournald (about 10% of I/O) and significant writes from pdflush:

1720(kjournald) 60460112
pdflush 31551
...

3. ext3 System Journaling

The kjournald daemon handles ext3 filesystem journaling. ext3 supports three journal modes:

journal : records both file data and metadata updates.

ordered (default): records only metadata updates and forces the log to be written before the metadata changes are committed.

writeback : records only metadata updates without forcing a write before the metadata change; the actual write timing depends on the OS flushing dirty pages.

4. pdflush and kswapd Daemons

pdflush synchronizes memory data to disk, writing dirty pages to ensure consistency between RAM and storage. kswapd manages memory, waking periodically to check for low free pages and releasing pages in batches when thresholds pages_low and pages_high are crossed.

5. Root Causes of IOWait

ext3 journaling (kjournald) generates a large amount of log data that must be flushed to disk by pdflush.

The filesystem is mounted with the default ordered mode and default mount options, offering no performance optimizations.

Relevant /etc/fstab entry:

LABEL=/ / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2

6. Disk Mount‑Option Optimization

Adding the noatime (or omitting nodiratime) mount option can improve disk read/write performance by 5‑10%.

LABEL=/ / ext3 defaults,noatime 1 1

Source: http://mdba.cn/2015/04/06/如何查找iowait元凶/