How I Restored a Production Server After Accidentally Deleting Everything with rm -rf

Accident Overview

A junior team member was asked to uninstall Oracle on a production server. She executed rm -rf $ORACLE_BASE/* without the $ORACLE_BASE variable set, which expanded to rm -rf /* and erased the whole filesystem, including Tomcat, MySQL, and other services.

Immediate Impact

The disk was mounted on another machine and inspected; all files were gone. Existing backups were corrupt – the latest usable backup was a 1 KB mysqldump from December 2013. The situation demanded an urgent data‑recovery effort.

First Rescue Attempt – ext3grep

The server used an ext3 filesystem, so the author downloaded and compiled ext3grep . After unmounting the volume to avoid overwriting deleted blocks, the following scan was run: ext3grep /dev/vgdata/LogVol00 --dump-names The command listed every deleted file and its path. A full restore was attempted: ext3grep /dev/vgdata/LogVol00 --restore-all Insufficient free space caused the restore to stop after only a few files. Individual files could be restored with:

ext3grep /dev/vgdata/LogVol00 --restore-file var/lib/mysql/aqsh/tb_b_attench.MYD

Automating MySQL File Recovery

All deleted filenames were dumped to a text file:

ext3grep /dev/vgdata/LogVol00 --dump-names > /usr/allnames.txt

MySQL‑related entries were filtered into mysqltbname.txt. A shell script then iterated over each line and restored the file, aborting on error:

while read LINE; do
  echo "begin to restore file $LINE"
  ext3grep /dev/vgdata/LogVol00 --restore-file $LINE
  if [ $? != 0 ]; then
    echo "restore failed, exit"
    # exit 1
  fi
done < ./mysqltbname.txt

The script ran for about 20 minutes and recovered roughly 40 MySQL files, far short of the ~300 files needed for all tables.

Second Tool – extundelete

The author also tried extundelete with directory restoration:

extundelete /dev/vgdata/LogVol00 --restore-directory var/lib/mysql/aqsh

This attempt failed, confirming that many files were already overwritten.

Alternative Recovery – MySQL Binlog

Since the MySQL service was configured to write binary logs, the author examined the available binlog files ( mysql-bin.000001, mysql-bin.000009, mysql-bin.000010). The first binlog could not be restored, but the larger mysql-bin.000010 succeeded:

mysqlbinlog /usr/mysql-bin.000010 | mysql -uroot -p

After supplying the password and waiting for the import to finish, the application came back online and most of the critical attendance data was recovered.

Post‑mortem Reflections

Never assign critical production tasks without clear communication of risks; always follow a change‑management process.

Automated backups must be verified regularly; a 1 KB dump is a clear sign of failure.

Implement real‑time monitoring and alerting for service anomalies to detect data loss early.

Avoid using the root account for routine operations; employ principle‑of‑least‑privilege users.

Through collective effort—team members staying late, the product manager keeping calm, and leadership providing support—the incident was resolved, but the experience underscored the importance of disciplined operations and robust backup strategies.