How I Rescued a Production Server After a Fatal rm -rf: A Step‑by‑Step Recovery Tale

Accident Background

During a routine Oracle installation on a production server, a junior colleague executed the command rm -rf $ORACLE_BASE/* without the $ORACLE_BASE variable set, turning it into rm -rf /*. The command, run as root, erased the entire filesystem, including Tomcat, MySQL databases, and other application files.

Realizing the severity, the team quickly unmounted the affected disk to prevent further writes and began searching for a recovery solution.

Lifeline – ext3grep

Research revealed ext3grep , a tool capable of recovering files deleted from ext3 filesystems. After installing it, the following steps were performed: ext3grep /dev/vgdata/LogVol00 --dump-names This command listed all deleted files and paths, confirming that many needed files were still recoverable.

Because the disk lacked sufficient free space, a full restore was impossible, so the team attempted selective restores:

ext3grep /dev/vgdata/LogVol00 --restore-file var/lib/mysql/aqsh/tb_b_attench.MYD

Only a handful of files were recovered. To automate the process, a script was written to restore every MySQL‑related file listed in a generated name file:

while read LINE; do
  echo "begin to restore file $LINE"
  ext3grep /dev/vgdata/LogVol00 --restore-file $LINE
  if [ $? != 0 ]; then
    echo "restore failed, exit"
    # exit 1
  fi
done < ./mysqltbname.txt

The script recovered about 40 files, far fewer than the ~300 files required for all MySQL tables.

Attempt with extundelete

The team also tried extundelete , which claims directory‑level restoration:

extundelete /dev/vgdata/LogVol00 --restore-directory var/lib/mysql/aqsh

This attempt failed, confirming that the deleted files were largely unrecoverable by this method.

Turning to MySQL binlog

Remembering that the MySQL service had binary logging enabled, the team located three binlog files ( mysql-bin.000001, mysql-bin.000009, mysql-bin.000010). After several failed attempts, the larger mysql-bin.000010 was successfully applied:

mysqlbinlog /usr/mysql-bin.000010 | mysql -uroot -p

The binlog import restored the missing attendance and mobile‑report data, and the application returned to normal operation.

After‑thoughts

Never assign critical production tasks without clear instructions and risk awareness.

Automated backups must be regularly verified; a 1 KB backup file is a red flag.

Implement real‑time monitoring and alerting for abnormal system behavior.

Never perform production changes as the root user; use principle‑of‑least‑privilege accounts.

The incident highlighted the importance of proper change management, reliable backups, and swift forensic recovery techniques.