How I Restored Accidentally Deleted Production Data with ext3grep and MySQL Binlog

Accident Background

A production server running Oracle and MySQL was unintentionally wiped when an uninstall script executed rm -rf $ORACLE_BASE/* as root. Because the ORACLE_BASE environment variable was unset, the command expanded to rm -rf /*, deleting the entire filesystem, including Tomcat, MySQL data directories, and other services.

Immediate Impact

The affected disk was detached and attached to another machine for forensic analysis. All application files were missing, and the only backup available was a 1 KB MySQL dump from December 2013, which was insufficient to restore the production system.

First Recovery Attempt – ext3grep

Using ext3grep, a tool that can recover deleted files from an ext3 filesystem, the volume was unmounted to prevent further writes and scanned: ext3grep /dev/vgdata/LogVol00 --dump-names The command listed every deleted file with its full path. An attempt to restore everything was made: ext3grep /dev/vgdata/LogVol00 --restore-all Disk‑space limitations caused the operation to abort after partially restoring about 40 files.

Targeted MySQL Restoration Script

To focus on MySQL data, the list of filenames was filtered for paths under var/lib/mysql and saved to mysqltbname.txt. A Bash loop restored each file individually:

while read LINE; do
  echo "begin to restore file $LINE"
  ext3grep /dev/vgdata/LogVol00 --restore-file "$LINE"
  if [ $? -ne 0 ]; then
    echo "restore failed, exit"
    # exit 1
  fi
done < ./mysqltbname.txt

The script ran for roughly 20 minutes and recovered dozens of .frm, .MYD, and .MYI files, but many tables remained missing.

Second Recovery Attempt – extundelete

Another tool, extundelete, was tried with a directory‑level restore:

extundelete /dev/vgdata/LogVol00 --restore-directory var/lib/mysql/aqsh

The command failed because the target blocks had already been overwritten.

Final Rescue – MySQL Binary Log Replay

The MySQL server had binary logging enabled. Three binlog files were located: mysql-bin.000001, mysql-bin.000009, and mysql-bin.000010. The smallest binlog could not be applied, but the largest ( 000010) was successfully replayed:

mysqlbinlog /usr/mysql-bin.000010 | mysql -uroot -p

After the import, MySQL started, the application came back online, and the missing attendance and performance data were recovered.

Post‑mortem Lessons

Never run destructive commands as root without explicitly defining all variables; always test on a non‑production copy first.

Maintain reliable, regularly verified backups; a 1 KB dump is not adequate for production recovery.

Implement monitoring and alerting to detect service anomalies early.

Enforce least‑privilege principles by creating dedicated, non‑root users for maintenance tasks.