When rm -rf Wiped a Production Server: Recovering MySQL Data with ext3grep and Binlog

Accident Background

A junior engineer was tasked with installing Oracle on a production server. While experimenting, she realized the installation was incorrect and attempted to uninstall it using a command that removed the Oracle base directory: rm -rf $ORACLE_BASE/* Because the $ORACLE_BASE variable was unset, the command became rm -rf /*, deleting the entire filesystem—including Tomcat, MySQL, and other services.

The mistake erased all data on the disk, leaving only a few large log files.

Initial Recovery Attempts

After discovering the loss, the team mounted the affected disk on another server and inspected the damage. Offline backups were only 1 KB and outdated, providing no usable data.

Remembering a previous case, the team tried ext3grep , a tool capable of recovering files from ext3 filesystems. ext3grep /dev/vgdata/LogVol00 --dump-names The command listed all deleted files, giving hope that the data could be recovered.

Since ext3grep cannot restore by directory, the team used the --restore-all option, but ran out of space on the target volume, resulting in partial restores. ext3grep /dev/vgdata/LogVol00 --restore-all They then redirected the file list to a text file, filtered MySQL‑related entries, and wrote a shell script to restore each file individually:

while read LINE
do
  echo "begin to restore file $LINE"
  ext3grep /dev/vgdata/LogVol00 --restore-file $LINE
  if [ $? != 0 ]; then
    echo "restore failed, exit"
    # exit 1
  fi
done < ./mysqltbname.txt

The script recovered about 40 files, far fewer than the ~300 MySQL files needed.

Using Binlog for Final Recovery

Realizing the remaining data might be in MySQL binary logs, the team located three binlog files ( mysql-binlog0001, mysql-bin.000009, mysql-bin.000010) and attempted to restore them.

mysqlbinlog /usr/mysql-bin.000010 | mysql -uroot -p

The binlog import succeeded, and the application data reappeared.

Post‑mortem and Lessons Learned

Never let an untrained person perform critical operations on a production system without clear instructions.

Ensure automated backups are verified and retained; a 1 KB backup is useless.

Implement monitoring and alerting to detect anomalies early.

Never operate as root for routine tasks; use least‑privilege accounts.

Through teamwork, the incident was resolved, but the experience highlighted the importance of proper procedures, reliable backups, and rapid recovery tools.