A Real-World Incident of Accidental Public Snapshot Sharing and Lessons Learned

In December 2018, the author was asked to share a disk snapshot of user A with user B and quickly logged into the server, completing the task in minutes.

Three minutes later he discovered the snapshot had been set to public = true , making it visible to all tenants and risking a serious data leak of an important customer.

Panicking, he performed a rollback, informed his team lead, and completed the rollback within five minutes, preventing further exposure.

The incident triggered alarms and inquiries from neighboring teams, exposing the lack of proper audit, review, and communication for high‑risk operations.

Reflecting on the root causes, the author notes the misuse of a simple public flag, treating high‑risk tasks as routine, and the absence of a productized UI for such operations.

He proposes concrete measures: isolate high‑risk APIs, enforce double‑check or approval workflows, avoid manual "human‑ops", document requirements, and push for product‑level tooling to handle risky actions safely.

The story concludes with a broader warning that engineers should not shoulder all risk alone and must demand proper safeguards and shared responsibility for critical cloud operations.