Linux rm -rf Disaster Recovery: From Incident Response to File System Forensics

Introduction

This article is a practical postmortem and hands-on manual for frontline operations engineers, not a popular science piece. It follows a real incident timeline from the moment of accidental deletion through tool preparation, disk protection, file system selection, recovery commands, risk assessment, verification, postmortem, and finally engineering practices to prevent future mis-deletions. All commands and parameters are written for common environments on RHEL/CentOS 7/8 and Ubuntu 20.04/22.04; behavior may vary across distributions, file systems, and kernel versions.

Chapter 1: Incident Background — How That rm -rf Was Executed

The incident occurred at 02:17 on a Tuesday. An ops engineer ("A") was cleaning an old backup server that received daily mysqldump outputs at 02:00 and rsynced them to a remote data center. The server had not been inspected for six months. The previous ops engineer left a script /opt/scripts/cleanup.sh containing:

find -L /data/backup -type d -mtime +30 -exec rm -rf {} \;

The -L flag followed symlinks. Engineer A restructured /data/backup into subdirectories ( mysql, app, logs, tmp) and changed the script path to /data/backup/tmp without code review or staging rehearsal. At 02:00 cron triggered the script. Because the previous engineer had created a symlink /data/backup/tmp -> /data/backup, find -L traversed the parent directory and recursively deleted mysql and app. The logs directory appeared to survive only because a background process held open file descriptors, keeping inodes alive temporarily. By 02:30 the remote rsync receiver alerted on three consecutive push failures, escalating to P1.

Chapter 2: First Response — Five Critical Steps

1. Preserve State, Do Not Reboot

Rebooting discards page cache and inode information held by running processes. Avoid reboot, init 6, shutdown -r now, sync (may trigger unexpected writes), and kill -9 on unrelated processes. Observe first, change nothing.

2. Isolate Write Traffic

Stop all processes writing to the affected filesystem:

systemctl stop crond
systemctl list-timers | grep -i mysql
ps -ef | grep -E "tar|rsync|mysqldump|java|python" | grep -v grep

If business cannot stop, mask the timer: systemctl mask crond (creates symlink to /dev/null).

3. Remove from Load Balancer / Scheduler Pool

Follow change management / emergency channel, synchronize, then act. Do not modify iptables without coordination.

4. Record Scene State

Collect: time/uptime/load ( date; uptime), disk usage ( df -h; df -i), memory ( free -h), process list ( ps auxf), mounts ( mount; cat /proc/mounts), open file descriptors ( lsof; lsof /data/backup), I/O stats ( iostat -dx 1 3; lsblk), filesystem type ( blkid; file -s /dev/sd*), fstab ( cp /etc/fstab /tmp/fstab_$(date +%Y%m%d_%H%M%S).bak), kernel logs ( dmesg; journalctl -k --since "1 hour ago"). Each item informs later recovery decisions.

5. Notify Stakeholders

P1 incident requires immediate notification to direct manager, application owner, DBA (if DB involved), InfoSec (if compliance data), backup system owner. Include time, impact scope, preliminary judgment, current actions, next steps.

Chapter 3: Why rm -rf Is Nearly Irreversible

1. Linux File Deletion Essence

Deleting a file does not zero disk bytes; it: (1) removes directory entry (dentry) from parent directory, (2) decrements link count, (3) if link count reaches zero, marks inode "unlinked" awaiting reclamation. Only when unlinked and no process holds the file (open count = 0) does the kernel return the inode to the bitmap and mark data blocks free for overwrite.

2. ext4 rm -rf Actions

VFS layer calls vfs_unlink / vfs_rmdir → ext4_unlink / ext4_rmdir → ext4_delete_entry (remove dentry) → ext4_dec_count (link count -1) → if file, ext4_free_inode_after_ordered marks inode free → marks corresponding blocks free. Blocks are not zeroed; data remains until overwritten.

3. xfs Differences

xfs uses B+ trees for inode (AGI) and block (AGF) management, dir2 format for directory entries. Deletion marks extents free quickly but does not zero. xfs log (xlog) aggressively writes metadata changes first. Recovery tools must parse complex on-disk structures; mainstream tool xfs_undelete has limited effectiveness compared to mature ext4 toolchain (extundelete, debugfs, testdisk).

4. Why Not 100% Recoverable

Blocks overwritten by new allocations

Journal replay modifying metadata

File truncated then rewritten (sparse + overwritten data)

Severe fragmentation scrambling block order

Filenames lost (many tools recover by inode only)

CoW filesystems (btrfs/zfs): snapshot exists → near 100% rollback; no snapshot → same luck as ext4/xfs

Recovered files may have names like inode_12345.dump, partial corruption, missing blocks in large files, or failed archive checksums. These risks must be communicated to business before attempting recovery.

5. Why Not Rely on "Trash"

Linux has no native trash; rm calls VFS unlink directly. Desktop environments (GNOME/KDE) provide ~/.local/share/Trash but servers rarely use them. trash-cli simulates macOS trash but has pitfalls: per-user, not cross-server, hard to retrofit legacy scripts, cannot prevent rm -rf on the trash directory itself. The incident resulted from three independent small issues (symlink + old script + path change) combining; fixing any one would have prevented it.

Chapter 4: File System Layer Principles — What Deleting a File Actually Touches

1. Disk to Filesystem Hierarchy

Physical disk /dev/sda
 -> Partition table (MBR/GPT)
    -> /dev/sda1, /dev/sda2 ...
       -> LVM PV / direct filesystem
          -> mkfs.ext4 / mkfs.xfs ...
             -> mount to /data
                -> User files / directories

Deletion occurs at "user files" layer; recovery tools operate at "filesystem" layer. Misalignment at any level causes failure.

2. ext4 Disk Layout

ext4 divides a partition into block groups (default size auto-calculated by mkfs.ext4). Core structures per group:

Superblock : FS metadata: block size, inode count, magic number

Group Descriptors : Describe each block group's location and size

Block Bitmap : Mark used blocks in this group

Inode Bitmap : Mark used inodes in this group

Inode Table : Store inodes (default 256 bytes each)

Data Blocks : Actual file data and directory entries

Key mkfs.ext4 parameters: -b 4096 (block size 4KB), -I 256 (inode size 256 bytes), -N 1000000 (reserve 1M inodes). These affect recovery tool parsing; extundelete auto-detects but may fail with journal corruption.

3. Inode Is the Recovery Key

Inode contains: i_mode (type+perm), i_uid / i_gid, i_size_lo / i_size_high, i_blocks_lo (block count), timestamps ( i_atime, i_ctime, i_mtime), i_dtime (deletion time — critical!), i_links_count, i_flags, i_block[15] (12 direct, 1 indirect, 1 double-indirect, 1 triple-indirect block pointers). If i_dtime != 0 but i_blocks > 0, the file is unlinked but blocks not yet reclaimed — recoverable. Small files (<48KB at 4KB block) fit in direct pointers; large files require parsing indirect blocks, explaining why recovered large files may have missing blocks.

4. ext4 Journal

Default journal enabled (unless -O ^has_journal). Metadata changes written to journal first, then asynchronously flushed. On delete: ext4_journal_start allocates handle → write inode/dentry changes to journal → ext4_journal_stop commits, journal flushed. Recovery tools: if journal not yet replayed, can read full delete trail; if replayed and overwritten, must scan for unlinked inodes.

5. xfs Disk Layout

xfs uses Allocation Groups (AG) each with AGF (free space bitmap), AGI (inode bitmap), AGFL (free inode list), Inode B+ Tree, Directory B+ Tree, and xfs log. Directory entries in B+ tree; deletion marks entry deleted but tree structure remains. Metadata complexity makes recovery tool implementation much harder.

6. btrfs / zfs "Native Protection"

CoW (Copy-on-Write) filesystems with native snapshots: btrfs subvolume snapshot /data /data/bak_20260609, zfs snapshot data/mysql@20260609. If recent snapshot exists, recovery success ~100%. Cloud providers (AWS EBS, Alibaba Cloud, Tencent Cloud) use CoW-style snapshots. The incident used ext4 without snapshots, forcing traditional toolchain.

Chapter 5: Immediate Mitigation — Preventing Recovery Actions from Causing Secondary Damage

1. Remount Deleted Partition Read-Only

df /data/backup
# assume /dev/sdb1
mount -o remount,ro /data
mount | grep /data  # should show (ro,...)

Requires no open write file descriptors; if EBUSY, use fuser -v /data to find writers. Safer: block-device read-only blockdev --setro /dev/sdb1 (freezes all FS on device, no writeback, no unmount required).

2. LVM Snapshot (Industrial-Grade Preferred)

vgdisplay | grep -E "VG Name|Free"
lvcreate -s -L 10G -n data_snap_20260609 /dev/vg0/data
mkdir -p /mnt/snap
mount -o ro /dev/vg0/data_snap_20260609 /mnt/snap
ls -la /mnt/snap/data/backup/mysql/

LVM snapshot uses COW: writes to origin LV copy original data to snapshot space; snapshot preserves point-in-time view. Safe to experiment recovery on snapshot. Cloud disks offer similar snapshot APIs (AWS create-snapshot, Alibaba/Tencent console/CLI).

3. Disk Image If No Remount or LVM

lsblk
dd if=/dev/sdb of=/dev/sdc bs=4M status=progress conv=noerror,sync
md5sum /dev/sdb; md5sum /dev/sdc

Risks: dd target mistake is catastrophic; double-check. Use oflag=direct to bypass page cache. conv=noerror,sync continues on bad sectors. Process saturates I/O; stop business first. Cloud VMs can use provider disk snapshot (COW-based, minimal impact).

4. Process-Held Files (Critical Detail)

lsof +L1 /data/backup

(link count ≤1) or lsof | grep deleted shows unlinked but open files. Example output:

mysqld  1234  mysql  8u  REG  253,1  1048576  12345  /data/backup/mysql/dump_20260608.sql (deleted)

Recover via /proc/$PID/fd/$FD:

ls -la /proc/1234/fd/8
cp /proc/1234/fd/8 /tmp/recovered_dump_20260608.sql

Lowest-cost recovery; must execute immediately before process exits.

5. Temporarily Stop syslog Writes

If syslog writes to the affected disk, its continuous writes overwrite free blocks. Stop rsyslog or redirect to tmpfs:

systemctl stop rsyslog
# or add to /etc/fstab: tmpfs /var/log tmpfs defaults,noatime,size=512M 0 0
mount -a

Chapter 6: Recovery Feasibility Assessment — 7-Step Environment Checklist

Confirm FS type: blkid /dev/sdb1, mount | grep /data, dumpe2fs -h /dev/sdb1.

Confirm mount point and options: cat /proc/mounts | grep /data, findmnt /data. Note relatime / strictatime / noatime affects timestamp accuracy.

Check ongoing writes: iostat -dx 1 5 — sustained w/s > 0 means active writes; stop them ASAP.

Disk health: smartctl -H /dev/sdb; smartctl -a /dev/sdb. Bad sectors drastically reduce success.

Disk space: df -h /data; df -i /data. Inode exhaustion prevents new file creation even with free space.

FS corruption: fsck -n /dev/sdb1 (ext, read-only), xfs_repair -n /dev/sdb1 (xfs). Diagnostic only.

Kernel logs: dmesg | tail -100, journalctl -k -p err --since "1 hour ago". Look for EXT4-fs error, I/O error, Buffer I/O error.

Chapter 7: Solution A — ext4 + extundelete (First Choice)

extundelete

(by ext2fsprogs maintainer) recovers specific files, entire directories, by inode scan, or by time window.

Critical Convention

All paths passed to extundelete are relative to filesystem root, NOT OS mount point. If /dev/sdb1 mounted at /data and file is /data/backup/mysql/dump.sql, pass /backup/mysql/dump.sql. Wrong path yields empty results without error.

Installation

# CentOS/RHEL (EPEL)
yum install -y epel-release
yum install -y extundelete
# Ubuntu/Debian
apt-get install -y extundelete
# Source compile if needed
wget https://sourceforge.net/projects/extundelete/files/extundelete/0.2.4/extundelete-0.2.4.tar.bz2
tar xf extundelete-0.2.4.tar.bz2
cd extundelete-0.2.4
./configure && make && make install

Preparation

mount -o remount,ro /data
# or umount /data
mkdir -p /mnt/recovery
mount /dev/sdc1 /mnt/recovery  # independent disk for output

Recover Specific File

extundelete /dev/sdb1 --restore-file /backup/mysql/dump_20260608.sql
# output in ./RECOVERED_FILES/

Recover Entire Directory

extundelete /dev/sdb1 --restore-directory /backup/mysql

Recover by Time Window

extundelete /dev/sdb1 --restore-all --before "2026-06-09 00:00:00"
# --after also available; combine to narrow range

Scan All Recoverable Files

extundelete /dev/sdb1 --list-all > /tmp/extundelete_list.txt 2>&1
extundelete /dev/sdb1 --restore-all

Common Pitfalls

Wrong path: Passing OS path ( /data/backup/...) yields empty RECOVERED_FILES/. Fix: run --list-all first, copy FS-relative paths from output.

Journal not found: can't find ext3 journal — version/kernel mismatch. Compile 0.2.4 with patches for newer kernels.

Recovered file size 0: i_size overwritten but blocks intact. Use debugfs to read by block range (see Chapter 8).

Filename becomes inode_NNNN : Normal; directory entries lost. Map back via inode.

Verification

find RECOVERED_FILES/ -type f | wc -l
ls -lS RECOVERED_FILES/ | head -20
file RECOVERED_FILES/*
md5sum RECOVERED_FILES/mysql/dump_20260608.sql  # compare with expected

Chapter 8: Solution B — ext4 + debugfs (Fine-Grained Recovery)

debugfs

(from e2fsprogs) is lower-level than extundelete. Use when extundelete fails, known inode, lost filenames, or severely damaged directory structure.

Enter debugfs Read-Only

debugfs -c /dev/sdb1  # -c = cat-like read-only mode
# or
debugfs -R "stats" /dev/sdb1 -c

Key Commands

debugfs: stat <12345>                    # show inode details
debugfs: dump <12345> /tmp/recovered_inode_12345  # extract file
debugfs: ls -l /data/backup/mysql      # list directory including deleted (shown as <12346>)
debugfs: lsdel                         # list all unlinked inodes with owner, size, dtime
debugfs: ncheck 12346                  # map inode to path (FS-relative)

Practical: Recover by Inode

# 1. Obtain inode from historical records (e.g., chat logs)
#    assume 12346 = dump_20260608.sql, 12347 = dump_20260607.sql
# 2. Recover
debugfs -c /dev/sdb1 -R "dump <12346> /mnt/recovery/dump_20260608.sql"
debugfs -c /dev/sdb1 -R "quit"

Practical: Recover All Files Under Deleted Directory

# 1. Find deleted directory inode (from lsdel, backup metadata, memory)
#    assume directory inode = 10001
echo "Assume directory inode = 10001"
# 2. Map inodes to paths
debugfs -c /dev/sdb1 -R "lsdel" | tee /tmp/lsdel.txt
debugfs -c /dev/sdb1 -R "ncheck 12346 12347 12348"
# output: 12346 /backup/mysql/dump_20260608.sql
# 3. Batch dump
for ino in 12346 12347 12348; do
  debugfs -c /dev/sdb1 -R "dump <$ino> /mnt/recovery/ino_$ino"
done

Common Issues

lsdel slow on large FS (>1T): traverses all inodes; check inode count first with dumpe2fs -h /dev/sdb1.

Dump size mismatch: i_size overwritten but blocks present. Read raw blocks via dd using block pointers from stat <inode>.

File type misidentified: first blocks overwritten. Read from tail of block range for identification.

Chapter 9: Solution C — xfs Recovery

xfs harder to recover but options exist.

1. xfs_undelete

yum install -y xfs_undelete  # or apt-get
xfs_undelete -t /data/backup/mysql /dev/sdb1
xfs_undelete -t /data/backup/mysql -o /mnt/recovery /dev/sdb1

Scans directory tree, not journal. Effectiveness depends on metadata overwrite. Test on snapshot first.

2. xfs_db (debugfs equivalent)

xfs_db /dev/sdb1
xfs_db> help
xfs_db> blockget
xfs_db> blockuse
xfs_db> quit

Primarily diagnostic/repair; limited recovery. Prefer xfs_undelete.

3. When All Else Fails

Immediate full-disk dd image

Engage professional data recovery firm

Assess business impact, activate standby

Postmortem: consider migrating to ext4 + LVM + snapshots

Many companies run backup servers on ext4 + LVM with daily snapshots — safer than xfs.

Chapter 10: Solution D — testdisk + photorec (Block-Level Blind Scan)

Part of sleuthkit; filesystem-agnostic, scans block content signatures.

Pros

No reliance on FS metadata

Supports ext, xfs, ntfs, fat, btrfs, zfs, hfs+

Good for small files, text, images

Cons

Slow

Filenames almost entirely lost

Directory structure lost

Suitable for "salvage scan", not precise recovery

Usage

yum install -y testdisk  # or apt-get
testdisk /dev/sdb1  # interactive: Create log -> select disk -> partition type -> Advanced -> Undelete
photorec /dev/sdb1  # interactive: select disk -> partition -> FS type (Other) -> output dir

photorec

produces thousands of f0001.jpg, f0002.txt requiring manual classification. Unsuitable for hundreds of GB of mysqldump SQL files (repetitive patterns cause fragmentation).

Chapter 11: Solution E — lsof Rescue of Unclosed Files (Lowest-Cost Recovery)

1. Find Deleted but Open Files

lsof | grep deleted
# output example (TID column presence varies by version):
mysqld  1234  1234  mysql  8u  REG  253,1  104857600  12345  /data/backup/mysql/dump_20260608.sql (deleted)
rsync   5678  5678  root   3r  REG  253,1  52428800  12346  /data/backup/app/app-20260608.tar.gz (deleted)

Columns: PID, FD (with access mode), size (bytes), inode, path + (deleted).

2. Recover Single File

cp /proc/1234/fd/8 /tmp/recovered_dump_20260608.sql
ls -la /tmp/recovered_dump_20260608.sql  # verify size matches

3. Batch Recovery Script (Uses lsof -F to Avoid Version-Dependent Column Count)

#!/bin/bash
# recover_deleted.sh
OUT_DIR="${1:-/tmp/recovered}"
mkdir -p "$OUT_DIR"
lsof -F pfn 2>/dev/null > /tmp/lsof_f.txt
awk -v OUT_DIR="$OUT_DIR" '
/^p/ { pid=substr($0,2); fd=""; name=""; deleted=0; next }
/^f/ { fd=substr($0,2); next }
/^n/ { name=substr($0,2); if (name ~ / \(deleted\)$/) { deleted=1; name=substr(name,1,length(name)-10) } }
{ if (deleted && pid!="" && fd!="" && name!="") { safe=name; gsub(///,"_",safe); target=OUT_DIR "/" pid "_" fd "_" safe; cmd="cp -a /proc/" pid "/fd/" fd " \"" target "\" 2>/dev/null"; if (system(cmd)==0) print "RECOVERED: " name " -> " target; deleted=0 } }
' /tmp/lsof_f.txt
rm -f /tmp/lsof_f.txt

Run:

chmod +x recover_deleted.sh; ./recover_deleted.sh /mnt/recovery

. Risks: brief FD contention, root required for other users' FDs, test first, handle spaces in filenames.

4. Practical Issues

Root-owned process holding mysql-owned file: recoverable (kernel checks caller credentials). /proc/$pid/fd missing: process already exited.

Script recovers fewer than lsof shows: some FDs closed but not yet reclaimed; retry with sleep or fall back to inode scan.

Chapter 12: Solution F — LVM / ZFS / btrfs Snapshot Rollback (Highest Success Rate)

1. LVM Snapshot

lvcreate -s -L 20G -n data_snap_recovery /dev/vg0/data
mount /dev/vg0/data_snap_recovery /mnt/snap
# if xfs snapshot in same VG causes UUID conflict:
# mount -o ro,nouuid /dev/vg0/data_snap_recovery /mnt/snap
cp -a /mnt/snap/data/backup/mysql/dump_20260608.sql /mnt/recovery/
diff /mnt/recovery/dump_20260608.sql /data/backup/mysql/dump_20260608.sql
umount /mnt/snap
lvremove -f /dev/vg0/data_snap_recovery

Limitations: snapshot space exhaustion invalidates it; heavy-write LVs unsuitable for many snapshots; snapshot impacts write performance.

2. btrfs Snapshot

btrfs subvolume list /data
btrfs subvolume snapshot /data /data/.snapshots/$(date +%F_%H%M%S)
mkdir -p /mnt/snap
mount -o subvol=.snapshots/2026-06-09_021800 /dev/sdb1 /mnt/snap
ls /mnt/snap/data/backup/mysql/
# or full restore
btrfs restore /dev/sdb1 /mnt/recovery_btrfs

Advantages: near-zero cost (COW), nested snapshots, incremental backup via btrfs send/receive.

3. zfs Snapshot

zfs list -t snapshot | grep data
zfs snapshot data@recover_$(date +%F_%H%M%S)
ls /data/.zfs/snapshot/recover_2026-06-09_021800/data/backup/mysql/
cp -a /data/.zfs/snapshot/recover_2026-06-09_021800/data/backup/mysql/* /mnt/recovery/
zfs destroy data@recover_2026-06-09_021800

Advantages: snapshots are first-class citizens; zfs rollback reverts entire FS; cross-host zfs send/receive for industrial backup.

4. Cloud Disk Snapshots

# AWS
aws ec2 create-snapshot --volume-id vol-0abc1234 --description "pre-recovery-$(date +%F)"
# Alibaba Cloud
aliyun ecs CreateSnapshot --DiskId d-abc1234 --Description "pre-recovery-$(date +%F)"
# Tencent Cloud
tccli cbs CreateSnapshot --DiskId disk-abc1234

Characteristics: COW-based, minimal I/O impact, seconds to create, cross-AZ replication for DR, pay per snapshot size + retention. After this incident, production migrated all ext4 volumes to LVM with daily snapshots retained 14 days.

Chapter 13: Real-World Timeline — Complete Misdeletion Recovery Walkthrough

Scenario: 2026-06-09 02:17, host backup-01.example.com (CentOS 7.9), deleted /data/backup/mysql (ext4), cause: script find ... -exec rm -rf followed symlink, ~200 mysqldump files, 380GB, local LVM snapshot from previous 02:00.

02:17-02:25: Response & Mitigation

ssh backup-01
sudo systemctl stop crond; sudo systemctl mask crond
ps -ef | grep -E "mysqldump|cleanup" | grep -v grep
sudo kill -STOP 5678  # pause cleanup.sh, don't kill yet
sudo sh -c 'date; uptime; df -h; df -i; free -h; mount > /tmp/mount.txt; lsof > /tmp/lsof.txt; lsblk > /tmp/lsblk.txt; blkid > /tmp/blkid.txt' > /tmp/initial_state.txt 2>&1
sudo mount -o remount,ro /data  # EBUSY
sudo fuser -vm /data
sudo kill -STOP $(pidof mysqldump)
sudo mount -o remount,ro /data  # success
mount | grep /data

02:25-02:30: LVM Snapshot

sudo vgdisplay vg0 | grep -E "VG Name|Free"  # Free: 50G
sudo lvcreate -s -L 30G -n data_snap_recovery /dev/vg0/data
sudo mkdir -p /mnt/snap
sudo mount -o ro /dev/vg0/data_snap_recovery /mnt/snap
sudo ls -la /mnt/snap/data/backup/mysql/ | head -20  # 380GB files present

02:30-02:50: lsof Rescue

sudo lsof | grep deleted | grep backup > /tmp/deleted_files.txt  # ~30 files held by mysqldump
sudo mkdir -p /mnt/recovery
sudo /opt/scripts/recover_deleted.sh /mnt/recovery  # recovered 30 files, ~80GB
sudo ls -la /mnt/recovery/ | head -20

02:50-04:30: extundelete Full Recovery

sudo mkdir -p /mnt/recovery2
sudo mount /dev/sdc1 /mnt/recovery2  # 1TB independent disk
sudo umount /mnt/snap  # keep snapshot LV
cd /mnt/recovery2
sudo extundelete /dev/sdb1 --restore-all --before "2026-06-09 02:00:00"  # ~1 hour
ls -la /mnt/recovery2/RECOVERED_FILES/ | head
ls -la /mnt/recovery2/RECOVERED_FILES/data/backup/mysql/ | head

04:30-06:00: Business Validation

for f in /mnt/recovery2/RECOVERED_FILES/data/backup/mysql/dump_*.sql; do
  head -100 "$f" | mysql -u root -p <db_test>
  if [ $? -ne 0 ]; then echo "BROKEN: $f"; fi
done
find /mnt/recovery2/RECOVERED_FILES/data/backup/mysql/ -type f -size -100k -ls
mysql -u root -p -e "SELECT * FROM <db>.tables" | wc -l  # compare INSERT row counts

06:00-08:00: Re-sync & Business Sign-off

rsync -avz /mnt/recovery2/RECOVERED_FILES/data/backup/mysql/ backup-target:/data/backup/mysql/
# Business confirms restored files work; missing 12 files pulled from remote DC
# Notify change completion

Postmortem (next morning)

Root cause: old script lacked code review

Underlying causes: no monitoring, no audit, no process

Improvements: all cleanup scripts require code review + dry-run

Chapter 14: Risk Scenarios & Unrecoverable Cases

Blocks overwritten: dumpe2fs /dev/sdb1 | grep -E "Free blocks|Free inodes" — low free blocks = high overwrite risk.

FS reformatted: mkfs.ext4 /dev/sdb1 resets metadata; only photorec block scan may salvage fragments.

Bad sectors:

smartctl -a /dev/sdb | grep -E "Reallocated|Pending|Uncorrectable"

— physical read failures yield zero/random blocks.

Partial overwrite: e.g., first 1MB of SQL file overwritten; recovered file fails checksum; must supplement from other sources.

Filenames completely lost: recovered as inode_12345.dump; map back via inode if recorded, else manual classification by size/time/type.

Encrypted FS (LUKS): need key; if key lost, data unrecoverable. Use key escrow, don't store in single password book, backup LUKS header offsite.

Write amplification: database checkpoint/redolog continuously overwrites free blocks; stop DB as early as possible.

Chapter 15: Prevention — Process, Commands, Backup, Monitoring

1. Process Layer

Code Review for cleanup scripts: any find ... -exec rm, rm -rf, shred must be in git, reviewed by ≥1 person, have dry-run option, use expiration logic (not just directory name).

Mandatory dry-run: script defaults to DRY_RUN=1 printing would-delete list; only DRY_RUN=0 executes real deletion.

Forced double-confirmation: wrapper script prompts for exact path, resolved path, disk usage, requires typing "YES".

Audit logging: alias rm to /usr/local/bin/audit_rm.sh logging timestamp, user, cwd, command to /var/log/rm_audit.log.

2. Command Layer

Replace rm with trash-cli : yum/apt install trash-cli or pip3 install trash-cli; alias rm='trash-put'. Caveats: per-user, per-host, habit transition period.

safe-rm: wrapper with blacklist ( /etc/safe-rm.conf includes /data/backup).

Prefer find -delete over find -exec rm : built-in, safer (no arbitrary command execution), smaller symlink attack surface.

Explicit globs: rm -rf /data/backup/*.sql after ls /data/backup/*.sql; avoid rm -rf /data/backup/*.

3. Backup Layer

3-2-1 strategy: 3 copies, 2 media types, 1 offsite. Implementation: local LVM snapshot, remote rsync, object storage (S3/OSS/COS).

Automated daily LVM snapshot script: creates data_daily_YYYYMMDD size 20G, retains 7 days, logs via logger.

Offsite backup:

rsync -avz --delete /data/backup/mysql/ [email protected]:/data/backup/mysql/

or

aws s3 sync /data/backup/mysql/ s3://my-bucket/mysql/ --delete

.

4. Monitoring Layer

Critical directory existence: cron script checks /data/backup/mysql, /data/backup/app, /data/backup/logs; alerts via HTTP POST if missing.

File count monitoring: Prometheus textfile collector exports backup_file_count{dir="/data/backup/mysql"}.

Cleanup script heartbeat: logs start/status to centralized logging (ELK/Loki).

Deletion event alerting: tail /var/log/rm_audit.log, regex for rm -rf, /, * patterns, immediate high-severity alert.

5. Configuration Layer

/etc/skel/.bashrc aliases:

alias rm='echo "Use trash-put or /opt/scripts/safe_rm.sh"; false'

, alias mv='mv -i', alias cp='cp -i' for new users.

/etc/profile.d/rm_alias.sh: forces alias rm='/usr/local/bin/audit_rm.sh' for all users (caution for root).

Chapter 16: Safe rm Alternatives

trash-cli: cross-platform, simple; doesn't solve symlink issue.

rmtrash (macOS): brew install rmtrash.

Quarantine directory: saferm moves targets to /var/spool/quarantine/YYYYMMDD_HHMMSS_path, auto-cleans after 30 days.

Git/Mercurial for configs/scripts: git commit -am "before change rm logic", git checkout HEAD -- cleanup.sh.

Git LFS / DVC for data files: lightweight version control for large datasets.

Chapter 17: Postmortem Summary & Advice

Key Findings

Direct cause: cleanup script no code review, no dry-run, symlink caused expanded deletion.

Root causes: no process (cleanup treated as trivia), no monitoring (17-min gap to alert), no audit (no double-confirm), no rehearsal (script change untested).

Impact: ~380GB invisible; remote backup existed but 1-day restore delay.

Action Items

All cleanup scripts under git + code review

Deploy audit_rm globally

Critical directory file-count monitoring + anomaly alerts

Migrate backup servers to ext4 + LVM + daily snapshots

Offsite backup encryption + integrity verification

Advice for Junior/Mid Ops

Always ls before rm -rf, even for your own scripts find -delete safer than find -exec rm Production cleanup scripts must support dry-run; rehearse in staging

Symlinks are rm -rf 's biggest accomplice; explicitly distinguish -type d vs -type l Monitor critical directories: existence + file count + total size

Annual real-data recovery drill

Never first-use a new tool in production; test first

Build muscle memory: first reaction to misdeletion is "snapshot", not "hands-on"

Advice for Team Leads

Cleanup tasks follow change process as strict as code deploy

Enforce code review; tighten git repo permissions

Quarterly "Anti-Misdeletion Day" drills

Code review checklist: symlinks, absolute paths, rm wrappers

Monitoring alerts must include "deletion event" category

Appendices

Appendix A: Command Cheatsheet

Tables for status checks ( df -h, df -i, mount, cat /proc/mounts, blkid, lsblk, iostat -dx 1 5, dmesg | tail, smartctl -H), recovery tools (extundelete, debugfs, xfs_undelete, xfs_db, testdisk, photorec, btrfs restore, lsof, zfs rollback), LVM ops, btrfs ops, zfs ops.

Appendix B: Common Error Codes

EBUSY (FS busy) — use fuser to find process, stop or unmount.

EACCES (permission) — run as root.

ENOSPC (output disk full) — use larger target disk.

EIO (disk read error) — add conv=noerror,sync to dd.

EINVAL (FS unrecognized) — check blkid .

ENOMEM (OOM) — increase memory or add swap.

Appendix C: Misconception Clarifications

sync

after rm cannot undo metadata changes.

Reformat + reboot doesn't guarantee data gone; photorec may still recover. rm -rf / behavior depends on mount topology; bind mounts can cause total loss.

xfs not unrecoverable; xfs_undelete and photorec work partially.

SSD TRIM reduces recovery; enterprise SSDs often disable/delay TRIM — check hdparm -I /dev/sdX | grep TRIM.

Recovery restores blocks; metadata (ctime, ACL, xattr) may be lost. dd vs rsync: different use cases (whole-disk vs filesystem-level). rm -rf ≠ rm; -f suppresses prompts and missing-file errors.

Appendix D: Extreme Fallback Options

Professional data recovery firms (costly, days/weeks, requires disk shipment, NDA).

Restore from "backup of backup": remote rsync, object storage snapshots, LTO tape (last resort, needs hardware).

Business degradation: disable non-critical features, run with partial data, gradual backfill.

Appendix E: Disk Imaging & Remote Recovery

# Local image
dd if=/dev/sdb of=/mnt/recovery/sdb.img bs=4M status=progress conv=noerror,sync
# Remote image
dd if=/dev/sdb bs=4M conv=noerror,sync | gzip | ssh user@backup "cat > /mnt/recovery/sdb.img.gz"
# Restore
dd if=/mnt/recovery/sdb.img of=/dev/sdb bs=4M status=progress
# ddrescue for bad media
ddrescue /dev/sdb /mnt/recovery/sdb.img /mnt/recovery/sdb.rescue.log
ddrescue -d -r3 /dev/sdb /mnt/recovery/sdb.img /mnt/recovery/sdb.rescue.log
# Loop-mount image for tool access
losetup -f /mnt/recovery/sdb.img
extundelete /dev/loop0 --restore-all

Appendix F: Drill Script (Production Use with Caution)

Create test image, populate data, record metadata, simulate rm -rf, practice mount -o remount,ro, extundelete, verify with diff. Success criteria: recovered files open correctly; failure: zero-byte or corrupted. Document time, commands, results into SOP.

Appendix G: Recommended Monitoring Metrics

backup_file_count{dir="/data/backup/mysql"}
backup_dir_size_bytes{dir="/data/backup/mysql"}
backup_last_modified_timestamp{dir="/data/backup/mysql"}
node_filesystem_avail_bytes{mountpoint="/data"}
node_filesystem_files_free{mountpoint="/data"}
lvm_snapshot_count{vg="vg0"}
remote_backup_last_sync_timestamp

Prometheus alert rules: BackupDirectoryMissing (count==0 for 5m), BackupDirectoryLowFileCount (<100 for 30m), BackupSyncFailed (last sync >24h).

Appendix H: Tool Comparison Matrix

extundelete : ext4 dir recovery, low difficulty, high success, first choice

debugfs : inode-level, medium, high, with metadata

xfs_undelete : xfs dir, low, medium, newer tool

testdisk : full FS scan, medium, medium, interactive

photorec : block signature, medium, medium-low, filenames lost

lsof : process-held, low, high, must-check

LVM snapshot : full rollback, low, 100%, needs snapshot

btrfs restore : full FS, medium, high

zfs rollback : full FS, low, 100%, needs snapshot

Appendix I: SOP Template

# Data Misdeletion Emergency Response SOP
## Trigger Conditions
- Deletion event alert received
- User reports missing files
- Monitoring shows sudden directory file count drop
## Response Steps
1. Confirm incident (within 10 min)
   - Contact reporter, SSH to target
2. Immediate mitigation (within 5 min)
   - Stop cron, related processes
   - Remount ro or create LVM snapshot
3. Scene recording (within 15 min)
   - Capture mount, lsof, ps, dmesg
4. Assess recovery plan (within 30 min)
   - Identify FS type, select tool
5. Execute recovery (time varies)
   - extundelete / debugfs / lsof
   - Write to independent disk
6. Business validation
   - File size, type, content
   - Application owner sign-off
7. Postmortem (within 24 hrs)
   - Write report, land action items

Appendix J: References & Tools

ext2fsprogs docs (debugfs, e2fsck, mke2fs)

e2fsprogs source (ext4 internals)

LVM official docs (snapshot internals)

BTRFS Wiki (CoW filesystem)

ZFS docs (zfs send/receive)

Tools: extundelete (sf.net), testdisk/photorec (cgsecurity.org), sleuthkit (sleuthkit.org), ddrescue (gnu.org), trash-cli (github.com/andreafrancia/trash-cli)

Conclusion

rm -rf

isn't scary; what's scary is "assuming backup exists so I'm safe". The longer you do ops, the more you respect "delete": unlike writes (git revert), it's like SQL DROP TABLE — once executed, it's gone. This article's true purpose isn't teaching extundelete but instilling: understand filesystem internals to respect deletion; think impact before acting; always have Plan B (backups, snapshots, code review); turn misdeletion prevention into process, tooling, and muscle memory. Technologies change (ext4→btrfs, CentOS→Rocky, rm→wrappers), but deletion's essence remains: irreversible, requires approval, requires backup. Hope this gives you more composure and confidence next time you face rm -rf.