How I Rescued a Production MySQL Database After a Fatal rm -rf Accident

Accident Background

A junior colleague was tasked with reinstalling Oracle on a production server. She ran rm -rf $ORACLE_BASE/* without the ORACLE_BASE variable set, which expanded to rm -rf /* and deleted the entire filesystem, including Tomcat, MySQL databases, and other services.

Immediate Response

The damaged disk was mounted on another machine for inspection. Backup files were found to be only 1 KB with outdated MySQL dump comments, and the latest reliable backup dated back to December 2013, leaving the system without a usable restore point.

Rescue Attempt with ext3grep

Searching online revealed ext3grep , a tool capable of recovering files from ext3 filesystems. After unmounting the affected volume to prevent further writes, the following command was used to list deleted files: ext3grep /dev/vgdata/LogVol00 --dump-names Since ext3grep cannot restore by directory, a full restore was attempted: ext3grep /dev/vgdata/LogVol00 --restore-all Disk space ran out, so selective restoration was performed. All deleted filenames were saved to /usr/allnames.txt and filtered for MySQL‑related files, producing mysqltbname.txt. A shell script then iterated over this list:

while read LINE; do
  echo "begin to restore file $LINE"
  ext3grep /dev/vgdata/LogVol00 --restore-file $LINE
  if [ $? != 0 ]; then
    echo "restore failed, exit"
    # exit 1
  fi
done < ./mysqltbname.txt

The script recovered about 40 files after ~20 minutes, but many MySQL table files (frm, myd, myi) remained missing.

Alternative Tool: extundelete

Next, extundelete was tried, which claims directory‑level restoration. The command used was:

extundelete /dev/vgdata/LogVol00 --restore-directory var/lib/mysql/aqsh

Unfortunately, the files were already corrupted and could not be recovered.

Turning to MySQL Binlog

Since the MySQL server had binary logging enabled, the binlog files ( mysql-bin.000001, mysql-bin.000009, mysql-bin.000010) were examined. Restoring from mysql-bin.000001 failed, but mysql-bin.000010 (several hundred MB) succeeded: mysqlbinlog /usr/mysql-bin.000010 | mysql -uroot -p After providing the root password, the import completed, and the application regained access to the previously lost data.

Post‑Incident Reflections

Never assign critical production tasks without clear risk communication and proper supervision.

Automated backups must be verified regularly; a 1 KB dump is a clear sign of failure.

Implement real‑time monitoring and alerting for abnormal disk activity or service failures.

Never perform destructive operations as the root user; use least‑privilege accounts.

Tools Referenced

ext3grep – https://code.google.com/p/ext3grep/

extundelete – http://extundelete.sourceforge.net/