What the Hearthstone Database Rollback Reveals About MySQL Backup Best Practices

Reflection on the Hearthstone Database Rollback Incident

Did you see the news about the Hearthstone database outage? The game suffered a server failure that caused data loss, leading the team to roll back to a previous state and compensate players.

Official announcement: http://hs.blizzard.cn/articles/16/8565

As a former DBA for online games, I have experienced similar rollback situations, but none as large as this.

DBA work is demanding.

Here are the issues identified from the announcement:

Announcement time : The incident was announced at 2017‑01‑18 18:00, with a rollback to 2017‑01‑14 15:20. The gap suggests a prolonged recovery attempt instead of an immediate rollback.

Power failure caused hardware damage. The MySQL setup likely lacked double‑write protection and used an old BBU, so a power loss easily corrupted cache data and the database.

Backup failure : The backup process also failed, possibly due to an incorrect character set in

mysqldump

and insufficient testing of backup restores.

Recommendations for server reliability and database backup:

Perform regular full backups , preferably physical, at least once daily.

Execute hourly incremental or differential backups, e.g., using binlog and a

last_update_time

column to limit data loss to under an hour.

Test backup restores regularly, randomly selecting backup sets to ensure every instance can be recovered.

Monitor hardware health (CPU, memory, RAID controller, battery, temperature) and set up alerts to prevent overheating‑induced reboots.