Recovering Data After an Accidental rm -rf on Linux: Step‑by‑Step Guide

Accident background

The incident occurred on a backup server at 02:17 AM when a newly edited cleanup.sh script executed find -L /data/backup -type d -mtime +30 -exec rm -rf {} \;. The script mistakenly pointed /data/backup to a soft link /data/backup/tmp -> /data/backup, causing rm -rf to traverse the parent directory and delete /data/backup/mysql, /data/backup/app and other sub‑directories.

Immediate response (first seconds)

1. Stop all cron jobs and related services:

systemctl stop crond
systemctl mask crond
systemctl stop mysqldump.service   # if running

2. Capture the current state for later analysis:

date; uptime; df -h; df -i; free -h; mount > /tmp/mount.txt; lsof > /tmp/lsof.txt; lsblk > /tmp/lsblk.txt; blkid > /tmp/blkid.txt

3. Remount the affected filesystem read‑only (or set the block device read‑only):

mount -o remount,ro /data
# or
blockdev --setro /dev/sdb1

4. Verify the mount is read‑only:

mount | grep /data

Why rm -rf is almost irreversible

Linux deletion removes the directory entry, decrements the inode link count and, only when the count reaches zero, marks the inode and its data blocks as free. The actual data blocks remain untouched until overwritten, which is why recovery is possible if the filesystem is not further written to.

File‑system specifics

ext4 : Deletion follows VFS → ext4_unlink → ext4_dec_count → ext4_free_inode_after_ordered. Blocks are merely marked free; no zeroing occurs.

xfs : Uses B+‑tree metadata; deletion clears directory entries but leaves block pointers. Recovery is harder because of the aggressive XFS journal.

CoW file systems (btrfs, zfs) : Snapshots provide near‑100 % recovery if a recent snapshot exists. Without snapshots, recovery falls back to the same block‑level techniques.

Recovery toolbox

extundelete

– primary tool for ext2/3/4, supports file, directory and time‑window restores. debugfs – low‑level inode inspection and dump. lsof – copy files still opened by processes (the cheapest recovery path).

LVM snapshots – lvcreate -s -L 20G -n data_snap /dev/vg0/data then mount read‑only. xfs_undelete – best‑effort recovery for XFS. testdisk / photorec – block‑level scan when metadata is lost.

Full disk imaging with dd or ddrescue for offline analysis.

Cloud‑disk snapshots (AWS, Alibaba, Tencent) – one‑click point‑in‑time copies.

Step‑by‑step recovery workflow

Capture environment : run the commands listed in the response section and store their output.

Make the filesystem immutable (remount ro or blockdev ro).

Live file rescue with lsof | grep deleted and the provided recover_deleted.sh script to copy any still‑open files.

Ext4 recovery :

# mount a clean target
mkdir -p /mnt/recovery
mount /dev/sdc1 /mnt/recovery
# run extundelete
extundelete /dev/sdb1 --restore-all --before "2026-06-09 02:00:00"

Use --restore-file or --restore-directory with paths relative to the filesystem root (e.g., /backup/mysql/dump.sql).

Inode‑level recovery with debugfs -c /dev/sdb1:

debugfs: lsdel
debugfs: dump <12345> /tmp/recovered_12345.bin

XFS : try xfs_undelete -t /data/backup/mysql /dev/sdb1 or use xfs_db for diagnostics.

Fallback block scan : run testdisk /dev/sdb1 or photorec /dev/sdb1 when metadata tools fail.

Verification : compare file counts, sizes, checksums, and optionally load SQL dumps into a test database.

Restore to production : rsync recovered files back, re‑enable services, and close the incident.

Post‑mortem : document the timeline, root cause (soft‑link + missing code review), and update SOPs.

Common pitfalls

Using absolute host paths with extundelete (the tool expects paths relative to the FS root).

Running extundelete after the journal has been replayed – the needed metadata may be gone.

Zero‑size files after recovery – often caused by overwritten blocks; use dd on the original inode’s block range.

Missing inode numbers – retrieve them via debugfs -R "lsdel" or extundelete --list-all.

Scenarios where recovery is impossible

Data blocks have been overwritten (high Free blocks usage, heavy I/O after deletion).

Filesystem re‑formatted (e.g., mkfs.ext4) – metadata is lost.

Bad sectors or hardware errors reported by smartctl.

Encrypted volumes without the key (LUKS header lost).

Prevention measures

Process level : all cleanup scripts must live in a Git repository, undergo code review, and provide a DRY_RUN=1 mode.

# example safe cleanup script
DRY_RUN=${DRY_RUN:-1}
if [ "$DRY_RUN" = "1" ]; then
  echo "DRY‑RUN: would delete …"
  find /data/backup/mysql -type f -mtime +30 -print
  exit 0
fi
# real deletion
find /data/backup/mysql -type f -mtime +30 -delete

Command wrappers : replace rm with trash‑cli or a custom safe‑rm script that requires explicit confirmation.

# safe‑rm wrapper example
#!/bin/bash
read -p "Type YES to delete $1: " ans
[ "$ans" = "YES" ] || { echo "Aborted"; exit 1; }
rm -rf "$1"

Audit logging : global alias to /usr/local/bin/audit_rm.sh that logs every rm with timestamp, user and PID.

Monitoring : Prometheus textfile exporter for critical directory existence, file count and size; alert on zero files or sudden size drop.

Backup strategy : 3‑copy, 2‑media, 1‑offsite rule; daily LVM snapshots retained 14 days; periodic rsync to remote host; object‑storage snapshots for immutable archives.

Snapshot policy : automate LVM snapshots via a cron job (example script provided) and retain only the needed number of snapshots.

Key takeaways

Understanding the inode‑based deletion process makes it clear why rm -rf is dangerous. Immediate isolation, thorough state capture, and the correct use of ext4‑specific tools ( extundelete, debugfs) can recover most data if performed quickly. For XFS or CoW file systems, dedicated tools or snapshots are required. Finally, embedding code review, dry‑run options, audit wrappers and robust monitoring into the operational workflow prevents accidental loss in the first place.