In this detailed post‑mortem, Alexey Grigorev recounts how using Claude Code to automate a Terraform deployment unintentionally erased his entire production environment and two‑and‑a‑half years of data, exposing the risks of over‑reliance on AI‑driven automation and highlighting essential safeguards.
A chronicle of real‑world operations incidents from 2003 to 2024 shows how simple mistakes—mis‑configured passwords, unplugged cables, hardware mix‑ups, and rushed cloud migrations—can cascade into massive outages, offering hard‑earned lessons for anyone managing production systems.
A collection of startling operational mishaps—from a disastrous database expansion during a sales event to a Kubernetes storage blunder, a misconfigured ESXi host, a company‑wide Excel crash, and a power‑maintenance disaster that fried servers—illustrates the critical importance of proper procedures, backups, and infrastructure monitoring.
This article recounts a series of shocking operational mishaps—including a Kubernetes PV/PVC deletion that erased an entire codebase, a careless shell script that killed the wrong processes, a rookie’s risky server formatting, and a mysterious Excel crash—highlighting the importance of proper backups, testing, and change control.
This article compiles six shocking operations incidents—from a Kubernetes PV/PVC deletion that erased an entire codebase, to a careless kill‑script that terminated critical services, a rookie admin formatting servers without backup, ESXi CPU saturation causing stock‑exchange timeouts, and a production DB expansion that wiped transaction data—highlighting the dire consequences of inadequate safeguards and the importance of rigorous operational practices.
The article analyzes a server outage triggered by a module that repeatedly created a scheduled task without proper lifecycle control, examines the problematic Java code, lists four key issues, presents a corrected implementation, and reflects on development, testing, review, and logging practices to prevent similar incidents.
An intern at ByteDance accidentally removed all sub‑GB machine‑learning models by deleting a parent directory with skip‑trash, prompting a P0 incident that sparked a massive internal discussion about model impact, flat‑management permissions, and the broader implications for AI operations.
During a night‑time outage, a XCache (Codis + Pika) instance hung due to massive write load triggering low‑level protection, causing Sentinel to switch masters; the proxy’s accept queue filled with timed‑out sockets, blocking new connections, so scaling the proxy layer and expanding capacity restored service while prompting automation, health‑check, and queue‑overflow alerts.
This postmortem details the July 13, 2021 Bilibili outage caused by a Lua‑induced CPU 100% bug in the OpenResty‑based SLB, describing the incident timeline, root‑cause analysis, mitigation steps, and the subsequent technical and process improvements to enhance reliability and multi‑active deployment.
An accidental change to make the HTTPie repository private caused GitHub to delete all its 54,000 stars and watches, and despite the author's attempts to restore them, GitHub refused, highlighting risks of repository mismanagement and prompting recommendations for clearer warnings and soft‑delete mechanisms.
A recent flash‑sale failure caused by a cache avalanche was analyzed, revealing that setting a uniform two‑hour expiration for all items flooded the database, and the post outlines detection steps, emergency mitigation, and three proven techniques—uniform expiration, mutex locking, and never‑expire caches—to prevent recurrence.
After a junior staff member mistakenly ran an unchecked rm‑rf command that erased an entire production server, the author details a step‑by‑step recovery using ext3grep, custom shell scripts, extundelete, and MySQL binlog replay, and concludes with lessons on backup, monitoring, and change management.
This article recounts several real incidents where disgruntled engineers or admins deleted critical databases as retaliation, highlighting the severe consequences and stressing that proper backups and cautious use of destructive commands are essential for any organization.
From Hearthstone’s dual‑database crash to Uber’s massive data breach, this 2017 Linux operations roundup chronicles twelve critical incidents—highlighting backup failures, Docker rebranding, ransomware, BGP hijacking, and more—offering key lessons for sysadmins and DevOps professionals.
