How I Survived an Accidental rm -fr /* Disaster and Restored My Server

Incident Overview

I was writing a simple Bash script for a holiday task and, due to a bug, the script executed rm -fr /* on an internal company server. The command started deleting the root filesystem, causing panic.

Root Cause Analysis

The variable new_lic_dir was assigned using back‑ticks, which caused the command substitution to fail and return an empty string. Consequently the deletion command became rm -fr /* instead of the intended rm -fr $new_lic_dir/*. The misuse of back‑ticks turned a harmless script into a destructive one.

Immediate Response

Realising the mistake, I immediately stopped the script with Ctrl+C. I kept the SSH session open and avoided rebooting the server, preserving the remaining environment for investigation.

Recovery Steps

Because ls was gone, I relied on cd with tab‑completion to navigate directories. I identified the four critical directories that were removed: /bin, /boot, /dev – completely deleted. /lib – dynamic libraries partially removed.

Using a second healthy server, I copied the missing /bin and /lib directories to a web server, then downloaded them with wget. Since chmod was unavailable, I granted execution permission via Perl: perl -e "chmod 777, 'ls'" Next, I packaged the original /bin directory into a tarball on the healthy machine, transferred it with wget, and extracted it on the damaged server using tar. The same process restored /boot, /dev, and the remaining parts of /lib.

Prevention Measures

To avoid accidental deletions, I implemented several safeguards:

Check directory before removal – only run rm -fr when the target variable is non‑empty:

#!/bin/bash
work_path=$(pwd)
if [ "${work_path}" != "" ]; then
  rm -fr ${work_path}/*
fi

Enable strict variable handling – add set -u to scripts so undefined variables cause an error and stop execution.

Replace rm with safe‑rm – configure /etc/safe‑rm.conf with a blacklist (e.g., /, /etc) to prevent dangerous deletions.

Implement a recycle‑bin mechanism – alias rm to a wrapper script that moves files to /home/.trash and schedules periodic cleanup via crontab.

Mount the root filesystem read‑only in /etc/fstab (using remount,ro) to make accidental rm ineffective.

Following these practices ensures that even if a destructive command is issued, the system either blocks it or provides a clear recovery path.

Key Takeaways

Never ignore warning signs; stop the command immediately.

Report the incident to leadership without concealment.

Keep the SSH session alive and avoid rebooting until recovery is complete.