Finding the Process That Writes a File on CentOS 7 with SystemTap

Background

On a CentOS 7 server the disk usage occasionally spikes to 99 %. Because the monitoring system only provides aggregated metrics, a manual snapshot of I/O statistics is needed. The required tools are iostat -dx -k for device queues, sar -u for CPU wait, and pidstat -d for per‑process I/O.

Steps

Generate a statistics file

cat>/tmp/at_task.sh<<EOF
pidstat -d 2 >/tmp/pidstat_`date +%F_%T`.log 2>& 1 &
sar -u 2 >/tmp/sar_`date +%F_%T`.log 2>& 1 &
while [ 1 ]; do echo -n `date +%T` >>/tmp/iostat_`date +%F` 2>& 1 && iostat -dx -k 1 1 >>/tmp/iostat_`date +%F` 2>& 1; sleep 2; done &
EOF

The date +%T command is added inside the while loop so each line of output is timestamped.

Schedule the script with

at

at 15:14 today -f /tmp/at_task.sh

Initial error: Can't open /var/run/atd.pid to signal atd. No atd running?

Solution: restart the atd service and re‑run the at command.

service atd restart

After restarting, the job runs and creates snapshot files.

Snapshot Output

Sample iostat output:

15:13:35Linux 3.10.0-862.14.4.el7.x86_64 (ip-xxxxx) 03/13/2019 _x86_64_ (4 CPU)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq‑sz avgqu‑sz await r_await w_await svctm %util
vda 0.12 0.07 17.31 19.41 580.79 90.52 36.57 0.09 2.39 4.42 0.57 0.72 2.63

Sample sar output:

03:14:00 PM CPU %user %nice %system %iowait %steal %idle
03:14:02 PM all 0.25 0.00 0.38 0.00 0.00 99.37

Sample pidstat output:

03:14:00 PM UID PID kB_rd/s kB_wr/s kB_ccwr/s Command
03:14:02 PM 5700 9089 0.00 6.00 0.00 uxxx

Attempt to kill the collecting processes:

ps -ef | egrep 'iostat|sar|pidstat|while' | grep -v grep | awk '{print $2}' | xargs -l kill

The while loop does not appear in the ps output, so it continues writing to /tmp/iostat_…. lsof also fails to locate the writer for that file, though it can find the process holding mysql‑error.log.

Finding the Writer with SystemTap

Install SystemTap: yum -y install systemtap SystemTap can trace inode writes. First obtain the inode of the target file:

stat -c '%i' /tmp/iostat_2019-03-13
# => 4210339

Identify the device’s major/minor numbers:

ls -al /dev/vda1
# brw-rw---- 1 root disk 253, 1 Jan 30 13:57 /dev/vda1

Run the inodewatch script (requires matching kernel‑devel and debuginfo packages):

stap /usr/share/systemtap/examples/io/inodewatch.stp 253 1 4210339

Initial attempts fail due to missing kernel-devel and version mismatches. Install the correct kernel-devel RPM and the corresponding kernel-debuginfo package, then retry.

wget ftp://.../kernel-devel-3.10.0-862.14.4.el7.x86_64.rpm
rpm -ivh kernel-devel-3.10.0-862.14.4.el7.x86_64.rpm
debuginfo-install kernel-3.10.0-862.14.4.el7.x86_64

If a module version mismatch persists, edit

/usr/src/kernels/3.10.0-862.14.4.el7.x86_64/include/generated/compile.h

to align UTS_VERSION with the running kernel, then clear SystemTap’s cache:

vim /usr/src/kernels/.../compile.h
# change #define UTS_VERSION "#1 SMP Tue Sep 25 14:32:52 CDT 2018"
# to "#1 SMP Wed Sep 26 15:12:11 UTC 2018"
rm -rf /root/.systemtap/cache/*
stap -v /usr/share/systemtap/examples/io/inodewatch.stp 253 1 4210339

After the fix, SystemTap reports lines such as: iostat(4671) vfs_write 0xfd00001/4210339 Thus the PID of the iostat process writing to the file is visible (e.g., 4671).

Correct Approach

Create a simpler script that runs the while loop directly and schedule it with at now + 1 minute. Then the PID can be obtained with a normal ps query:

cat>/tmp/iostat.sh<<EOF
while [ 1 ]; do echo -n `date +%T` >>/tmp/iostat_`date +%F` 2>& 1 && iostat -dx -m 1 1 >>/tmp/iostat_`date +%F` 2>& 1; sleep 2; done &
EOF
at now + 1 minute
echo "bash /tmp/iostat.sh"
ps -ef | grep iostat

This method reliably shows the writer’s PID (e.g., root 8593 1 0 16:16 pts/2 00:00:00 bash /tmp/iostat.sh).