Recovering Accidentally Deleted Production Server Data with ext3grep, extundelete, and MySQL Binlog

During a routine Oracle installation on a production server, a junior staff member executed rm -rf $ORACLE_BASE/* (or, when the variable was unset, rm -rf /* ) as root, unintentionally erasing the entire filesystem, including Tomcat, MySQL databases, and other services.

After discovering the loss, the team mounted the affected disk on another server, inspected the damage, and found that offline backups were either missing or only 1 KB in size, leaving no usable restore point.

Searching for a solution, they discovered the ext3grep tool, which can recover files deleted from ext3 filesystems. They unmounted the volume to prevent further writes and ran ext3grep /dev/vgdata/LogVol00 --dump-names to list all deleted files, then attempted a full restore with ext3grep /dev/vgdata/LogVol00 --restore-all . Disk space constraints caused partial failures.

To automate recovery of specific MySQL files, they redirected the file list to /usr/allnames.txt , filtered MySQL‑related entries into mysqltbname.txt , and executed a shell loop that invoked ext3grep /dev/vgdata/LogVol00 --restore-file $LINE for each entry, ultimately recovering about 40 files.

Because many MySQL table files remained missing, they tried extundelete with extundelete /dev/vgdata/LogVol00 --restore-directory var/lib/mysql/aqsh , but the files were too corrupted to restore.

Finally, they turned to MySQL binary logs. By locating three binlog files (mysql‑binlog0001, mysql‑bin.000009, mysql‑bin.000010) they replayed the most recent log using mysqlbinlog /usr/mysql-bin.000010 | mysql -uroot -p , which successfully restored the missing attendance and mobile‑report data.

The post concludes with a reflective “post‑mortem” listing four key lessons: the necessity of clear change procedures, reliable automated backups, real‑time monitoring with alerts, and avoiding root for routine operations.