How to Pinpoint High Disk I/O Sources on Linux MySQL Servers

In routine operations, administrators often encounter high server I/O load (high util in iostat) without being able to pinpoint the responsible processes or files.

This situation is especially common with MySQL; before performance_schema in MySQL 5.6, it was difficult to know which file caused the load.

This article presents a simple workflow to locate the source of high I/O.

Tools needed

iotop – http://guichaz.free.fr/iotop/

pt-ioprofile – http://www.percona.com/downloads/percona-toolkit/2.2.1/

Step 1: Use iostat to examine I/O

Run iostat -x 1 and observe which disk (e.g., dfa) shows high utilization, then focus on that device.

Step 2: Identify the responsible process with iotop

iotop is a Python script that reads thread I/O information from /proc and aggregates it. The screenshot shows most I/O coming from the mysqld process, confirming the load originates from the database.

Step 3: Find the specific file using pt-ioprofile

pt-ioprofile attaches strace to a given PID and records I/O activity per file. The tool is intrusive and should not be used on production servers unless you accept the risk (see warning below).

However, it works by attaching strace to the process using ptrace(), which will make it run very slowly until strace detaches. In addition to freezing the server, there is also some risk of the process crashing or performing badly after strace detaches, or indeed of strace not detaching cleanly and leaving the process in a sleeping state. As a result, this should be considered an intrusive tool, and should not be used on production servers unless you are comfortable with that.

Find the MySQL process ID with ps aux|grep mysqld, then run pt-ioprofile on that PID. By default the tool shows I/O time per file.

For more useful analysis, display I/O throughput using the --cell=sizes option, which reports results in B/s.

The output reveals that the main I/O source is sbtest (the I/O‑bound OLTP test from sysbench), with pressure concentrated on reads.