What Happens When a Production Database Is Accidentally Deleted? Lessons from GitLab’s Disaster

1. Introduction

On the fifth day of the Chinese New Year, a GitLab.com operator mistakenly deleted the production database, exposing the entire incident timeline and prompting a deep dive into the causes and consequences.

2. Event Review

GitLab initially posted a Google Doc with the incident details and later published a blog post. A colleague (YP) was performing load‑balancing work on the online database when a DDoS attack caused usage spikes. After blocking attacker IPs, a staging database (db2.staging) lagged 4 GB behind the primary.

Attempting to fix the sync, YP found the staging database hung and tried to delete it to start a fresh replica. The delete command was mistakenly run against the production cluster (db1.cluster), wiping the primary database.

"Work overtime" + "switching between many terminal windows" led to the mistake.

During recovery, only the db1.staging database was usable; five other backup mechanisms failed:

No database sync webhook.

No disk snapshot backup.

pg_dump used the wrong version (9.2 vs 9.6), producing no dump.

S3 backup was missing.

Backup scripts and documentation were unreliable and manual.

Even if these backups worked, they run only once per day, so data loss is inevitable without real‑time replication.

The restored data was six hours old, resulting in loss of thousands of projects, forks, imports, commits, users, and recent webhooks.

External experts, such as 2nd Quadrant’s CTO Simon Riggs, suggested improvements:

Investigate PostgreSQL 9.6 replication hang bugs.

Accept that a 4 GB replication lag is normal.

Reduce max_wal_senders to 2‑4 and max_connections to ~2000.

Use pg_basebackup, repmgr, barman (with S3 support), and regularly test backup/restore procedures.

GitLab’s staff lacked deep PostgreSQL expertise.

GitLab opened several related issues to address the shortcomings (e.g., updating PS1 prompts, adding Prometheus monitoring, adjusting PostgreSQL settings, implementing point‑in‑time recovery, hourly LVM snapshots, Azure disk snapshots, moving staging to ARM, automating backup testing, improving documentation).

3. Related Thoughts

3.1 Technical Perspective

1) Manual Operations

Directly executing commands on production is a poor habit; strong operational capability correlates with automated, auditable change processes. Code should manage machines, not humans.

Adding more people to a task makes it labor‑intensive; automation should replace manual effort.

Complex permission systems or checklists often become additional maintenance burdens without solving root causes.

Data loss can stem from power failures, disk crashes, malware, etc.; procedural safeguards cannot prevent all scenarios.

2) Backup Strategies

Backups are typically periodic, so any failure incurs data loss between the last backup and the incident. Version incompatibilities and dormant disaster‑recovery sites further complicate restoration.

High‑availability systems require continuously live, multi‑node architectures.

AWS S3 offers 99.999999999% durability, ensuring data survives hardware failures and site outages.

Even with perfect backups, human error can still cause loss; robust, live replication is essential.

3.2 Non‑Technical Perspective

1) Post‑mortem Practices

Serious incidents should trigger thorough post‑mortems, such as Amazon’s “5 Whys” analysis, to uncover root causes.

2) Engineer Culture

Technical organizations should trust technology to solve problems rather than relying on policies and procedures alone.

3) Transparency

Openly sharing incident details, as GitLab and AWS do, builds trust and helps the community learn from failures.