How a Single AI‑Driven Command Wiped 2.5 Years of Production Data

Migration plan and shared Terraform configuration

The static site AI Shipping Labs (originally hosted on GitHub Pages) was to be migrated to AWS S3, with DNS moved to Route 53 and a new Django implementation deployed on a sub‑domain. To minimise effort, the existing Terraform configuration that manages the DataTalks.Club production environment (VPC, ECS cluster, load balancers, bastion host) was reused, so the new project would share the same AWS resources.

Cost‑saving decision and its side effects

By adding the new project to the existing Terraform state, the estimated monthly saving was only $5‑$10, but the two projects now shared a single VPC, network, and ECS cluster. This coupling meant that any Terraform change would affect both workloads.

Missing Terraform state leads to unintended resource creation

On 26 February, a deployment was triggered via the AI‑assisted tool Claude Code. The operator skipped the usual terraform plan review and ran terraform apply directly. Because the local terraform.tfstate file was absent (it remained on a previous workstation), Terraform assumed a clean environment and began creating a full set of resources: a new VPC, subnets, security groups, ECS services, and related cloud components.

Attempted cleanup and the fatal terraform destroy

After noticing the unexpected creation, the operator used the AWS CLI (through Claude) to list and delete the newly created duplicate resources, preserving the original production stack. Claude then suggested running terraform destroy to clean up “the resources it created”. The command was executed without realizing that the current Terraform state now represented **all** resources (both original and duplicate). Consequently, terraform destroy removed the genuine production infrastructure as well.

Full production outage

The destroy operation erased the RDS database instance, VPC, ECS cluster, load balancers, and bastion host, wiping two‑and‑a‑half years of course data (assignments, rankings, and other records).

Backup failure and emergency AWS support

Daily RDS snapshots appeared missing in the console; attempts to open the backup event failed. An AWS support ticket was opened, and the support tier was upgraded (incurring ~10 % higher cloud cost). AWS support discovered a hidden snapshot stored internally that was not visible in the console.

Recovery over 24 hours

The operator rebuilt the infrastructure with Terraform, simplifying the architecture (e.g., consolidating load balancers). Using the hidden snapshot, the RDS instance was restored, and the courses_answer table was verified to contain 1,943,200 rows. The platform was brought back online within a day.

Post‑incident safeguards

Disabled Claude Code’s automatic file writes and command execution; AI now only generates terraform plan output for manual approval.

Implemented multi‑layer backup strategy independent of Terraform lifecycle: automated RDS snapshots, S3‑based logical backups, and enabled RDS deletion protection.

Added automated backup verification – a replica is restored daily and a sanity‑check query is run to ensure data integrity.

The incident demonstrates that while AI‑driven automation can accelerate workflows, critical safety checks (state management, manual plan review, independent backups) must remain under human control.