When a Snapshot Leak Triggered a P0 Outage: Lessons on Manual Cloud Ops

One night I received a request from a teammate in the data group to share user A's disk snapshot with user B. I logged into the production server and completed the operation in about two minutes.

Three minutes later, while opening a new console page, I noticed the snapshot flag public = true. I panicked when I realized the snapshot had been shared with every tenant, meaning the data of an important customer could be accessed by anyone.

My heart raced and I trembled as I debated whether to roll back silently. I ultimately told my team lead, who urged me to roll back immediately. I edited the SQL, rolled back the snapshot, and after five minutes confirmed that no tenant had used the snapshot, so no actual data was leaked.

Background: Two months earlier a major client demanded data sharing between two tenants. Several complex solutions were discussed, but I thought the cloud‑management system could simply share resources by creating and sharing a snapshot. I was proud of the quick delivery, even though I lacked deep storage knowledge and the product team was not involved. Using an open‑source framework, I added the feature rapidly but ignored a “nuclear‑level” risk: setting public = true would expose the resource to all users.

Initially I assumed this would be a low‑frequency, one‑off operation. However, the client began using it repeatedly, and I grew complacent, executing the high‑risk command in minutes without thorough checks, which eventually led to the incident.

Deep reflections and lessons learned:

Technical solutions must go beyond basic resource‑sharing APIs; they need a visual interface and product‑level capabilities.

High‑risk APIs should be isolated into dedicated endpoints, never mixed with ordinary functions.

Critical online operations must have an audit mechanism or a double‑check process.

If no operational UI exists, push for a product feature or, at minimum, document every request and keep a personal record of the effort.

When a low‑frequency task becomes frequent, shift the responsibility and risk back to the product and management teams, making them aware that manual ops are unsafe and must be productized.

In short, never shoulder all the pressure and risk of high‑danger manual operations alone; advocate for proper tooling, audits, and clear responsibility to protect both the system and yourself.