How I Rescued a Production MySQL Database After a Fatal rm -rf Disaster

Accident Overview

A junior engineer executed rm -rf $ORACLE_BASE/* on a production server while the $ORACLE_BASE variable was unset. The command expanded to rm -rf /* and, running as root, removed the entire filesystem, including Tomcat, MySQL data directories and other critical files.

The damaged disk was mounted on a separate machine for analysis; only a 1 KB backup file existed, and the live system had been running for months without a reliable backup.

Recovery with ext3grep

After unmounting the volume to prevent further writes, ext3grep (available at https://code.google.com/p/ext3grep) was used to scan the ext3 filesystem. ext3grep /dev/vgdata/LogVol00 --dump-names To attempt a full restore: ext3grep /dev/vgdata/LogVol00 --restore-all Disk‑space limitations caused many restores to fail, so individual MySQL files were targeted:

ext3grep /dev/vgdata/LogVol00 --restore-file var/lib/mysql/aqsh/tb_b_attench.MYD

A shell script automated the restoration of all MySQL‑related files that had been listed in mysqltbname.txt:

while read LINE; do
    echo "begin to restore file $LINE"
    ext3grep /dev/vgdata/LogVol00 --restore-file $LINE
    if [ $? != 0 ]; then
        echo "restore failed, exit"
        exit 1
    fi
done < ./mysqltbname.txt

The script recovered roughly 40 files in about 20 minutes, far fewer than the ~300 MySQL table files (each table has .frm, .MYD, .MYI files).

Attempt with extundelete

The tool extundelete (http://extundelete.sourceforge.net) was tried for directory‑level recovery:

extundelete /dev/vgdata/LogVol00 --restore-directory var/lib/mysql/aqsh

The command failed, indicating that the data blocks had already been overwritten.

Recovery Using MySQL Binary Logs

The MySQL server had binary logging enabled. The available binlog files were:

mysql-binlog0001

mysql-bin.000009

mysql-bin.000010

The largest binlog ( mysql-bin.000010) was applied to the database:

mysqlbinlog /usr/mysql-bin.000010 | mysql -uroot -p

After importing the binlog, the application returned to service and most critical data were restored.

Key Technical Takeaways

Always unmount a filesystem before using low‑level recovery tools to avoid further data loss. ext3grep can list deleted inode names ( --dump-names) and restore individual files ( --restore-file), but full‑volume restores may be limited by available space. extundelete may fail when the target blocks have been overwritten; it does not guarantee directory restoration.

Enabling MySQL binary logging provides a reliable fallback for point‑in‑time recovery when file‑level restores are insufficient.

Automating file‑by‑file restoration with a simple shell loop can speed up the recovery process, but expect incomplete results if many files have been overwritten.