Identify the Process Writing to a File on CentOS 7 with SystemTap and Linux Monitoring Tools

Background

On a CentOS 7 server disk usage sometimes spikes to 99 %. The monitoring system only provides aggregated metrics, so per‑process I/O must be captured manually. The required snapshots are obtained with iostat -dx -k, sar -u and pidstat -d at regular intervals.

Procedure

1. Create a script that writes the snapshots

cat >/tmp/at_task.sh <<'EOF'
pidstat -d 2 >/tmp/pidstat_$(date +%F_%T).log 2>&1 &
sar -u 2 >/tmp/sar_$(date +%F_%T).log 2>&1 &
while true; do
    echo -n $(date +%T) >>/tmp/iostat_$(date +%F) && \
    iostat -dx -k 1 1 >>/tmp/iostat_$(date +%F) 2>&1
    sleep 2
done &
EOF

2. Schedule the script with at

at 15:14 today -f /tmp/at_task.sh

If atd is not running, start it:

service atd restart

3. Collected files

After the job finishes you will have files such as:

15:13:35 Linux 3.10.0-862.14.4.el7.x86_64 (host) 03/13/2019 _x86_64_ (4 CPU)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq‑sz avgqu‑sz await r_await w_await svctm %util
vda 0.12 0.07 17.31 19.41 580.79 90.52 36.57 0.09 2.39 4.42 0.57 0.72 2.63

Similar sar and pidstat logs are also generated.

4. Why lsof does not show the writer

The loop recreates /tmp/iostat_YYYY‑MM‑DD continuously, so each write uses a new inode. lsof can only display processes that hold a single, stable inode, therefore it returns no result for the file.

5. Use SystemTap to trace the inode

Install SystemTap and the required kernel development packages:

yum -y install systemtap
# Install matching kernel‑devel and debuginfo
yum -y install kernel-devel-$(uname -r)
debuginfo-install kernel-$(uname -r)

Obtain the inode number of the target file:

stat -c '%i' /tmp/iostat_2019-03-13
# example output: 4210339

Find the major/minor numbers of the underlying block device:

ls -l /dev/vda1
# output contains “253, 1” → major=253, minor=1

Run the SystemTap example script inodewatch.stp with those parameters:

stap /usr/share/systemtap/examples/io/inodewatch.stp 253 1 4210339

If the kernel‑devel version does not match the running kernel, edit /usr/src/kernels/$(uname -r)/include/generated/compile.h and change the UTS_VERSION string to the current kernel version, then clear the SystemTap cache:

vim /usr/src/kernels/$(uname -r)/include/generated/compile.h
# modify the line:
#define UTS_VERSION "#1 SMP Wed Sep 26 15:12:11 UTC 2018"
rm -rf /root/.systemtap/cache/*

After the script runs successfully it prints lines such as: iostat(4671) vfs_write 0xfd00001/4210339 This confirms that the iostat process is writing to the file and shows its PID.

6. Recommended way to launch the monitoring loop

Place the loop in a dedicated script so the PID can be obtained and the process stopped cleanly.

cat >/tmp/iostat.sh <<'EOF'
while true; do
    echo -n $(date +%T) >>/tmp/iostat_$(date +%F) && \
    iostat -dx -m 1 1 >>/tmp/iostat_$(date +%F) 2>&1
    sleep 2
done &
EOF

Schedule it with at (or start it directly):

at now + 1 minuteash /tmp/iostat.sh
# Verify the PID
ps -ef | grep iostat

Now the PID is visible with ps -ef | grep iostat and can be terminated when the monitoring is no longer needed.

Reference

Original article: https://www.cnblogs.com/YangJiaXin/p/10531197.html