Identify the Process Writing to a File on CentOS 7 with SystemTap and Standard Monitoring Tools

Background

On a CentOS 7 server disk usage reached 99 %. To capture a snapshot of I/O, CPU and per‑process statistics the following tools are used: iostat -dx -k – device queues, await, svctm, %util sar -u – %iowait, %user pidstat -d – per‑process I/O

Step 1 – Create a statistics‑collection script

cat >/tmp/at_task.sh <<'EOF'
pidstat -d 2 >/tmp/pidstat_$(date +%F_%T).log 2>&1 &
sar -u 2 >/tmp/sar_$(date +%F_%T).log 2>&1 &
while true; do
    echo -n $(date +%T) >>/tmp/iostat_$(date +%F) 2>&1
    iostat -dx -k 1 1 >>/tmp/iostat_$(date +%F) 2>&1
    sleep 2
done &
EOF

The while loop prefixes each iostat line with a timestamp so the data can be correlated later.

Step 2 – Schedule the script with at

at 15:14 today -f /tmp/at_task.sh

If the atd daemon is not running, start it first:

service atd restart
at 15:14 today -f /tmp/at_task.sh

Step 3 – Sample output

Typical excerpts from the generated logs:

iostat

15:13:35 Linux 3.10.0-862.14.4.el7.x86_64 (host) 03/13/2019 _x86_64_ (4 CPU)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda   0.12   0.07 17.31 19.41 580.79 90.52 36.57   0.09   2.39   4.42   0.57 0.72 2.63

sar

03:14:00 PM CPU %user %nice %system %iowait %steal %idle
03:14:02 PM all   0.25   0.00   0.38    0.00   0.00 99.37

pidstat

03:14:00 PM UID   PID  kB_rd/s kB_wr/s kB_ccwr/s Command
03:14:02 PM 5700 9089   0.00    6.00    0.00    uxxx

Problem – Identifying the writer process

The file /tmp/iostat_YYYY-MM-DD is created by a background while loop, so lsof does not show an open descriptor. The goal is to obtain the PID that holds the file’s inode.

Step 4 – Use SystemTap inodewatch script

Install SystemTap and required kernel headers:

yum -y install systemtap kernel-devel-$(uname -r) debuginfo-install kernel-$(uname -r)

Find the inode of the target file:

stat -c '%i' /tmp/iostat_2019-03-13
# => 4210339

Determine the major/minor numbers of the underlying block device (e.g. /dev/vda1):

ls -l /dev/vda1
# => brw-rw---- 1 root disk 253,1 /dev/vda1

Run the example script with those parameters (the script is located at /usr/share/systemtap/examples/io/inodewatch.stp):

stap /usr/share/systemtap/examples/io/inodewatch.stp 253 1 4210339

If a “module version mismatch” error occurs, edit /usr/src/kernels/$(uname -r)/include/generated/compile.h to make the UTS_VERSION string match the running kernel, then clear SystemTap’s cache:

vim /usr/src/kernels/$(uname -r)/include/generated/compile.h
# change the line defining UTS_VERSION to the current kernel’s version string
rm -rf /root/.systemtap/cache/*

After the script runs successfully it prints lines such as: iostat(4671) vfs_write 0xfd00001/4210339 This confirms that the iostat process (PID 4671) is writing to the file.

Step 5 – Simpler reliable method

Create a dedicated script that only runs the iostat loop, schedule it with at, and obtain its PID directly:

cat >/tmp/iostat.sh <<'EOF'
while true; do
    echo -n $(date +%T) >>/tmp/iostat_$(date +%F) 2>&1
    iostat -dx -m 1 1 >>/tmp/iostat_$(date +%F) 2>&1
    sleep 2
done &
EOF

at now + 1 minute -f /tmp/iostat.sh
# After the job starts:
ps -ef | grep iostat
# Example output:
root 8593 1 0 16:16 pts/2 00:00:00 bash /tmp/iostat.sh

This approach avoids SystemTap entirely and provides the writer’s PID in a straightforward way.