Recovering Data After Accidental rm -rf on Linux: A Complete Incident Response Guide

Incident Background

A backup server running CentOS 7 suffered accidental deletion of critical directories (/data/backup/mysql, /data/backup/app) at 02:17 due to a cleanup script using

find -L /data/backup/tmp -type d -mtime +30 -exec rm -rf {} \;

. The script followed a symlink /data/backup/tmp -> /data/backup, causing recursive deletion of parent directories. Only /data/backup/logs survived because files were held open by a process.

Immediate Response (First Seconds)

1. Preserve State, Do Not Reboot

Rebooting discards page cache and inode references held by processes. Avoid reboot, init 6, sync, and killing unrelated processes.

2. Isolate Write Traffic

systemctl stop crond
systemctl mask crond
ps -ef | grep -E "tar|rsync|mysqldump|java|python" | grep -v grep

3. Remove from Load Balancer

Follow change management process to drain traffic.

4. Record Scene State

date; uptime; df -h; df -i; free -h
mount > /tmp/mount_$(date +%Y%m%d_%H%M%S).txt
lsof > /tmp/lsof_$(date +%Y%m%d_%H%M%S).txt
iostat -dx 1 3 > /tmp/iostat_$(date +%Y%m%d_%H%M%S).txt
blkid > /tmp/blkid_$(date +%Y%m%d_%H%M%S).txt
dmesg > /tmp/dmesg_$(date +%Y%m%d_%H%M%S).txt

5. Notify Stakeholders

Inform manager, app owner, DBA, security, backup owner with time, impact, initial assessment, actions, next steps.

Why rm -rf Is Nearly Irreversible

Linux File Deletion Essence

Remove directory entry (dentry) from parent directory.

Decrement link count.

If link count reaches 0 and no open file descriptors, inode is freed and blocks marked free.

Data remains on disk until overwritten.

ext4 Behavior

vfs_unlink -> ext4_unlink -> ext4_delete_entry -> ext4_dec_count -> ext4_free_inode_after_ordered -> mark blocks free

Blocks are not zeroed.

XFS Differences

Uses B+ trees for inodes and blocks; metadata changes logged aggressively. Recovery tools (xfs_undelete) are less mature.

Why 100% Recovery Is Impossible

Blocks overwritten by new data.

Journal replay modifies metadata.

Truncate then rewrite creates holes.

Severe fragmentation scrambles block order.

Filenames lost; recovery yields inode-based names.

CoW filesystems (btrfs/zfs) need snapshots.

Filesystem Internals

ext4 Disk Layout

Block groups contain superblock, group descriptors, block bitmap, inode bitmap, inode table, data blocks. Key mkfs parameters: -b 4096 -I 256 -N 1000000.

Inode Structure

struct ext4_inode {
  __le16 i_mode;
  __le32 i_size_lo;
  __le32 i_atime, i_ctime, i_mtime, i_dtime; // deletion time critical
  __le16 i_links_count;
  __le32 i_block[15]; // direct, indirect, double, triple indirect
};

If i_dtime != 0 and i_blocks > 0, file is unlinked but blocks not reclaimed.

ext4 Journal

Metadata changes written to journal first. If journal not replayed, recovery tools see full delete trail.

XFS Layout

Allocation Groups (AG) with AGF, AGI, AGFL, inode B+ tree, directory B+ tree, xfs log.

btrfs/zfs Snapshots

btrfs subvolume snapshot /data /data/bak_20260609
zfs snapshot data/mysql@20260609

Near 100% recovery if snapshot exists.

Containment: Prevent Secondary Damage

Remount Read-Only

mount -o remount,ro /data
# or blockdev --setro /dev/sdb1

LVM Snapshot (Recommended)

lvcreate -s -L 10G -n data_snap_20260609 /dev/vg0/data
mount -o ro /dev/vg0/data_snap_20260609 /mnt/snap

Disk Image (dd)

dd if=/dev/sdb of=/dev/sdc bs=4M status=progress conv=noerror,sync

Rescue Open Files via /proc

lsof +L1 /data/backup
lsof | grep deleted
cp /proc/1234/fd/8 /tmp/recovered_dump.sql

Stop syslog Writes

systemctl stop rsyslog
# or mount tmpfs on /var/log

Environment Assessment Checklist

Confirm FS type: blkid /dev/sdb1 Mount options: cat /proc/mounts (note noatime)

Disk write activity: iostat -dx 1 5 Disk health: smartctl -a /dev/sdb Space/inodes: df -h /data; df -i /data FS integrity: fsck -n /dev/sdb1 or xfs_repair -n Kernel logs:

dmesg | tail -100

Recovery Method A: extundelete (Primary for ext4)

Install

yum install -y epel-release && yum install -y extundelete
# or apt-get install -y extundelete

Prepare

mount -o remount,ro /data
mkdir -p /mnt/recovery && mount /dev/sdc1 /mnt/recovery

Restore File

extundelete /dev/sdb1 --restore-file /backup/mysql/dump_20260608.sql

Path is relative to filesystem root, not mount point.

Restore Directory

extundelete /dev/sdb1 --restore-directory /backup/mysql

Time Window

extundelete /dev/sdb1 --restore-all --before "2026-06-09 00:00:00"

List All

extundelete /dev/sdb1 --list-all > /tmp/list.txt

Common Pitfalls

Wrong path (include mount prefix) yields empty result.

Journal not found: version mismatch; compile from source.

Recovered file size 0: inode size overwritten; use dd with block range.

Filenames become inode_NNNN.dump.

Verify

find RECOVERED_FILES/ -type f | wc -l
ls -lS RECOVERED_FILES/ | head -20
file RECOVERED_FILES/*
md5sum RECOVERED_FILES/mysql/dump_20260608.sql

Recovery Method B: debugfs (Fine-Grained)

Open Read-Only

debugfs -c /dev/sdb1

Key Commands

stat <12345>
dump <12345> /tmp/recovered_inode_12345
ls -l /data/backup/mysql
lsdel
ncheck 12346

Recover by Inode

debugfs -c /dev/sdb1 -R "dump <12346> /mnt/recovery/dump_20260608.sql"

Batch Restore Deleted Directory

debugfs -c /dev/sdb1 -R "lsdel" | tee /tmp/lsdel.txt
for ino in 12346 12347 12348; do
  debugfs -c /dev/sdb1 -R "dump <$ino> /mnt/recovery/ino_$ino"
done

Troubleshooting

lsdel

slow on large FS; check inode count with dumpe2fs -h.

Wrong file size: i_size corrupted; read raw blocks via dd.

File type misidentified: read tail blocks for signature.

Recovery Method C: XFS

xfs_undelete

yum install -y xfs_undelete
xfs_undelete -t /data/backup/mysql -o /mnt/recovery /dev/sdb1

xfs_db

Diagnostic tool; limited recovery.

If Unrecoverable

Create full disk image.

Engage professional recovery service.

Activate standby systems.

Consider migrating to ext4 + LVM snapshots.

Recovery Method D: testdisk / photorec (Block-Level Scan)

Filesystem-agnostic, recovers by file signatures. Slow, loses filenames and directory structure. Good for heterogeneous small files, last resort.

Recovery Method E: lsof (Lowest Cost)

Recover files still held open by processes.

lsof -F pfn 2>/dev/null > /tmp/lsof_f.txt
# parse with awk script (provided in article) to copy from /proc/$pid/fd/$fd

Recovery Method F: Snapshots (Highest Success)

LVM

lvcreate -s -L 20G -n data_snap_recovery /dev/vg0/data
mount -o ro /dev/vg0/data_snap_recovery /mnt/snap
cp -a /mnt/snap/data/backup/mysql/... /mnt/recovery/

btrfs

btrfs subvolume snapshot /data /data/.snapshots/$(date +%F_%H%M%S)
mount -o subvol=.snapshots/2026-06-09_021800 /dev/sdb1 /mnt/snap

ZFS

zfs snapshot data@recover_$(date +%F_%H%M%S)
cp -a /data/.zfs/snapshot/recover_.../data/backup/mysql/* /mnt/recovery/

Cloud Disk Snapshots

aws ec2 create-snapshot --volume-id vol-xxx --description "pre-recovery-$(date +%F)"

Real-World Timeline (Case Study)

02:17-02:25 Response & Containment

ssh backup-01
systemctl stop crond; systemctl mask crond
kill -STOP $(pidof cleanup.sh)
mount -o remount,ro /data  # after stopping writers

02:25-02:30 LVM Snapshot

lvcreate -s -L 30G -n data_snap_recovery /dev/vg0/data
mount -o ro /dev/vg0/data_snap_recovery /mnt/snap
ls -la /mnt/snap/data/backup/mysql/

02:30-02:50 lsof Rescue

lsof | grep deleted | grep backup > /tmp/deleted_files.txt
./recover_deleted.sh /mnt/recovery  # recovered 30 files, 80GB

02:50-04:30 extundelete Full Restore

cd /mnt/recovery2
extundelete /dev/sdb1 --restore-all --before "2026-06-09 02:00:00"
# ~1 hour

04:30-06:00 Business Validation

Load dumps into test DB, verify row counts, checksum critical files.

06:00-08:00 Sync & Verify

Rsync recovered files to production; app team confirms restore; missing 12 files pulled from remote backup.

Unrecoverable Scenarios

Blocks overwritten (check dumpe2fs | grep Free blocks).

Filesystem reformatted ( mkfs).

Bad sectors ( smartctl -a shows reallocated/pending).

Partial overwrite (head/tail corruption).

Filenames completely lost.

Encrypted FS without key (LUKS).

Write amplification from running database.

Prevention: Policy, Commands, Backup, Monitoring

Policy

All destructive scripts in Git, mandatory code review, dry-run flag, expiration logic.

Wrapper scripts enforce DRY_RUN=1 default, require YES confirmation.

Audit logging via alias rm=/usr/local/bin/audit_rm.sh.

Command Layer

Use trash-cli or safe-rm with blacklist.

Prefer find -delete over find -exec rm.

Avoid wildcards; list first.

Backup Layer

3-2-1 strategy: local LVM snapshot, remote rsync, object storage.

Automated daily LVM snapshot script with retention.

Monitoring Layer

Directory existence checks.

File count/size metrics via Prometheus textfile collector.

Cleanup script heartbeat logging.

Real-time audit log alerting on dangerous patterns.

Config Layer

/etc/skel/.bashrc

and /etc/profile.d/rm_alias.sh enforce safe aliases.

Safe Deletion Alternatives

trash-cli

(cross-platform recycle bin). rmtrash (macOS).

Quarantine directory with auto-expiry ( saferm script).

Git/Mercurial for configs/scripts.

DVC/Git LFS for large data.

Postmortem & Recommendations

Root Causes

Script changed without code review or dry-run.

Missing monitoring (17 min gap).

No audit/confirmation for rm.

No staging rehearsal.

Action Items

All cleanup scripts in Git with review.

Deploy audit_rm globally.

Add directory monitoring + alerts.

Migrate backup server to ext4 + LVM daily snapshots.

Encrypt remote backup + integrity checks.

Advice for Junior/Mid Ops

Always ls before rm -rf.

Use find -delete not find -exec rm.

Scripts must support dry-run.

Symlinks are rm -rf amplifiers; handle explicitly.

Monitor critical directories (existence, count, size).

Annual real recovery drills.

Never first-use new tools in production.

Instinct: snapshot first, act later.

Advice for Team Leads

Treat cleanup tasks as formal changes.

Enforce code review via Git permissions.

Quarterly "anti-deletion" drills.

Review checklist includes symlinks, absolute paths, rm wrappers.

Alerting must have "deletion event" category.

Appendices

Include command cheat sheets (status, recovery tools, LVM, btrfs, ZFS), error code troubleshooting, common misconceptions (sync after rm, SSD TRIM, dd vs rsync), extreme fallback (professional recovery, backup-of-backup, business degradation), disk imaging with dd / ddrescue, drill playbook, Prometheus alerting rules, tool comparison matrix, SOP template, recommended reading.