How a Simple System‑Time Change Sparked a Massive Outage

A new operations engineer at an internet company was asked by a receptionist to correct a billing system that was running a year behind. Without thinking, he changed the Linux system time by one year, assuming it would not affect the billing system.

Shortly after the change, all online users disappeared. Customer support received hundreds of calls about network outages, and the monitoring system raised large‑scale disconnection alerts. The engineer realized a serious incident was occurring.

The team leader, "Tao", was alerted and rushed to the site with other ops engineers and DBAs. They discovered that the time shift caused all accounts that should have expired within the next year to be considered expired, kicking more than 3,000 users offline and preventing them from logging back in.

The DBA wrote a SQL statement to extend the expiration dates of those accounts to the end of the year, backed up the relevant tables, and executed the fix. The operation took over 40 minutes, after which the affected users could log in again and the service was restored, though the billing records no longer matched the financial statements, resulting in a loss of over 400,000 yuan.

The incident was classified as a Level‑1 severe accident caused by human error. The engineer was demoted, lost his bonus, and underwent a three‑month reassessment. His manager also received a serious disciplinary record.

Comments from other readers highlighted poor permission management, the danger of giving front‑desk staff direct access to production systems, and similar mishaps such as accidental database deletions, network debug commands that took a core switch offline, and a short‑circuit that blacked out an entire base station.

The article concludes by urging discussion on how to improve process, policy, and permission management to avoid such incidents.