How a Nighttime Hot‑Key Surge Overwhelmed a Database Server—and the Ops Fixes That Saved It

Incident Overview

On a Wednesday night, a DBA named Xiao Dong received an alarm that the server 10.25.125.31 experienced a network traffic surge, with the internal NIC traffic jumping from 300 Mbps to 1 Gbps, hitting the NIC limit.

Multiple RTX groups reported a drop in module call success rates to 98% and increased read/write timeouts, indicating that the overloaded storage server was affecting dozens of business services.

Root Cause Analysis

Experience suggested the spike was due to either a "fat" key (very large record) or a hot key (frequently accessed record). Monitoring pinpointed a hot business ID (BID) in the life‑service table generating over 100 reads per second.

Further log analysis revealed several hot keys each around 700 KB, with one key alone consuming roughly 560 Mb of traffic per second.

Mitigation Steps

The team attempted to migrate other business data off the overloaded server, but the NIC was saturated, so they first throttled the life‑service traffic, which raised migration speed to 100 Mbps and allowed moving 50 GB of data in about an hour.

After traffic normalized, they identified the hot keys, confirmed they belonged to a popular public account used in a marketing campaign, and decided on immediate actions.

Optimization Measures

Move the identified hot keys to a dedicated 10 Gbps storage machine.

Implement a one‑minute cache at the application layer for hot keys to reduce backend load.

Write a script to scan backup data, calculate the proportion and average size of fat keys, and report to developers.

Work with product to split oversized records (e.g., limit each record to 100 KB and split larger ones).

Infrastructure Upgrade

Using the “Weave Cloud” operations platform, Xiao Dong deployed a pair of 10 Gbps storage machines from the warehouse, migrated the hot key records within 20 minutes, and verified the deployment.

Automation Ideas

He also drafted ideas for a periodic backup file scanner to detect unreasonable fat keys and proposed an automatic diagnosis and self‑healing workflow that would trigger migration tools when hot‑key‑induced traffic spikes are detected.

Final Outcome

During the next marketing burst, the newly provisioned 10 Gbps storage handled traffic up to 3.4 Gbps without issues, confirming the effectiveness of the mitigation and optimization steps.