Bizarre Ops Disasters: From K8s Data Loss to Accidental Process Killing

When the author first joined a small company that had aggressively moved everything onto Kubernetes, all persistent storage was mounted via a single NFS-backed PV and PVC, with sub‑directories separating services. To decommission a service, they attempted to delete its PVC, but the operation mistakenly cleared the entire shared directory, wiping all source code. Fortunately, a month‑old backup taken during a migration allowed a full rollback.

Another incident involved a colleague’s shell script designed to kill processes matching certain criteria. The script used ps | grep | xargs kill without extracting only the PID column, so every field of the matching line was passed to kill. This unintentionally killed a critical middleware container whose command line contained numbers that matched the script’s pattern, triggering a cascade failure under heavy traffic. The script was executed with:

chmod +x script
./script

A third story describes a newly hired operations engineer who was tasked with formatting and reinstalling the OS on one of four Dell servers. Despite repeated reminders to back up the code repository on the third server, the engineer proceeded to format it without a backup. In this case, a prior backup existed, so no data was lost, but the incident underscored the risks of ignoring basic backup procedures.

Additional anecdotes include a securities firm that experienced timeouts during market peaks despite low pod and node utilization; the root cause was an ESXi host whose CPU was pegged at 100 % after a BIOS‑version maintenance migration that created an uneven VM distribution. Rebalancing the VMs resolved the issue. Another quirky case involved an office-wide Excel crash that was mysteriously resolved by a simple reboot, restoring all documents.

These real‑world examples serve as cautionary tales for operations teams, emphasizing the need for thorough testing, precise scripting, reliable backup strategies, and vigilant resource monitoring.