How I Rescued a Production Server After Accidentally Deleting All Files

Accident Background

A junior colleague was tasked with reinstalling Oracle on a production server. While following an online uninstall guide, she executed rm -rf $ORACLE_BASE. Because the ORACLE_BASE variable was unset, the command expanded to rm -rf /*, deleting the entire filesystem—including Tomcat, MySQL data, and other applications.

The server hosted a customer‑facing production system that had been running for many months, so an urgent recovery was required.

Rescue Attempt with ext3grep

Searching online revealed ext3grep , a tool capable of recovering files deleted with rm -rf on ext3 partitions. The disk was unmounted to prevent further writes, ext3grep was compiled and installed, and a scan for deleted filenames was run: ext3grep /dev/vgdata/LogVol00 --dump -names The tool listed all deleted files, but could only restore everything at once: ext3grep /dev/vgdata/LogVol00 --restore-all Due to insufficient free space, only a subset of files could be restored. Individual files were restored with commands such as:

ext3grep /dev/vgdata/LogVol00 --restore-file var/lib/mysql/aqsh/tb_b_attench.MYD

All deleted MySQL filenames were dumped to a file and processed by a shell script that iterated over each line and invoked ext3grep to restore it.

while read LINE; do
  echo "begin to restore file $LINE"
  ext3grep /dev/vgdata/LogVol00 --restore-file $LINE
  if [ $? != 0 ]; then
    echo "restore failed, exit"
    # exit 1
  fi
done < ./mysqltbname.txt

About 40 files were recovered, far fewer than the ~300 MySQL files needed.

Trying extundelete

Another tool, extundelete , was tested with the hope of directory‑level restoration:

extundelete /dev/vgdata/LogVol00 --restore-directory var/lib/mysql/aqsh

The attempt failed; the files were already corrupted.

Turning to MySQL Binlogs

Remembering that MySQL binlogs were enabled, the author located three binlog files (e.g., mysql-bin.000010) and attempted to restore them. The first binlog failed, but the larger mysql-bin.000010 succeeded:

mysqlbinlog /usr/mysql-bin.000010 | mysql -uroot -p

After loading the binlog, the missing attendance and mobile‑report data reappeared.

Post‑mortem and Lessons Learned

Never assign critical production tasks without clear instructions and proper supervision.

Automated backups must be verified; a 1 KB dump file indicated a broken backup process.

Implement real‑time monitoring and alerting to detect failures early.

Never perform destructive operations as the root user; use least‑privilege accounts.

The incident highlighted the importance of disciplined change management, reliable backups, and collaborative problem solving.

Tools referenced:

ext3grep: https://code.google.com/p/ext3grep/

extundelete: http://extundelete.sourceforge.net/