What Caused the Massive P1 Outage? A Real‑World Security Scanning Bug Uncovered

Rarely do we encounter a fault that forces the technical lead to become invisible in meetings, leaving only progress reports; this situation is intolerable because problems are the engine of team growth.

Yesterday afternoon the company’s leadership was bombarded with calls: merchants could not log in, single sign‑on was down, users could not perform any actions, and the impact was massive.

Investigation quickly revealed that all user passwords had been reset at the same moment. The updateTime field in the database showed that the operation originated from business logic rather than a DBA, because DBA‑made changes would only appear in the binlog.

Further digging showed that the passwords were not identical after the reset; each was random, which ruled out a simple UPDATE statement. The root cause turned out to be a security engineer who had recently added a weak‑password verification feature to a scanning engine. The engine repeatedly attempted logins, hit the retry limit, and triggered a bug that reset every password.

The security engineer reported the issue, the DBA extracted the relevant records from the binlog and generated the necessary SQL to revert the changes, and the problem was resolved.

This incident exposed the fragility of the system and underscored the importance of a dedicated security team, proper coordination, and disciplined incident‑response practices.

It also demonstrated that even well‑intentioned security tools can become the source of catastrophic failures if not carefully designed and monitored.