10 AM System Auto‑Paid ¥600K: A Full Post‑mortem of a Commission Settlement Incident

Incident Trigger

At 10 am the alert group erupted when a monitoring alarm reported that the day's commission settlement total reached seven times the historical same‑day amount, spiking from the usual ¥200 K to ¥600 K without any manual operation or scheduled activity; the batch job ran as scheduled.

Immediate Damage Control

The technical lead ordered “close withdrawals” and within ten minutes the withdrawal channel was shut down, the settlement batch paused, and downstream fund‑outflow interfaces were circuit‑broken, completing the stop‑loss in under twenty minutes.

Close user withdrawal entry to prevent abnormal funds from leaving.

Pause settlement batch to block further erroneous processing.

Freeze related order status to buy time for data rollback.

Any delay would have increased fund outflow and recovery cost.

Root‑Cause Investigation

The team traced the settlement batch SQL call chain and pinpointed a conditional logic error: a recent bug‑fix omitted a status check in an else if branch, causing orders that should have remained “settling” to be mistakenly marked as “ready for settlement”. This branch covered about 80 % of active orders.

The missing check originated from a rushed urgent fix for another settlement delay issue; the code review missed the edge‑case branch, and test data did not simulate high‑concurrency order volumes, so the bug only manifested under production load.

1. Business requirement change triggered code modification.

2. Bug fix rushed, review process compressed.

3. Boundary condition not covered by tests.

4. Production data characteristics differed from test environment.

None of these steps alone was fatal, but together they caused the incident.

Emergency Fix and Data Rollback

After locating the issue, developers added the missing status validation, ran regression tests in the pre‑release environment, and deployed the fix urgently.

Rolling back the ¥600 K of erroneous commissions involved generating a detailed exception list, marking those records invalid, and restoring original order states. The SQL rollback went through three versions: the first missed foreign‑key constraints, the second ignored lock contention, and the third added batch commits with row‑level locks, finally succeeding after thorough testing.

The rollback spanned nine hours, from 6 pm to 3 am, because many operations cannot run during normal business hours and must not conflict with other batch jobs.

Post‑mortem Review

The team held a retrospective without blame, focusing on three recurring questions:

Why did static monitoring thresholds miss earlier detection? Fixed thresholds become ineffective as business scales; a sudden 7× jump triggered the alarm, but a 2× increase might have gone unnoticed.

Why was the review process compressed for the bug fix? Emergency pressure led to “deploy first, review later”, accumulating hidden risk.

Why didn’t tests cover the branch? Test data were based on historical samples and never reproduced a scenario with massive concurrent settlements.

Evolution: Business Monitoring Platform

Two months later the team launched a business monitoring platform with three core capabilities:

Real‑time data warehouse : settlement data latency reduced from T+1 to seconds, enabling instant metric queries.

Ratio‑based dynamic thresholds : instead of fixed limits, the system computes a baseline from historical ratios and triggers alerts and circuit‑breaks when values deviate beyond a configurable multiple.

Visual dashboards : key metrics such as settlement amount, order volume, and user distribution are displayed with drill‑down capabilities.

The design principle is to intercept anomalies at the moment they occur rather than discovering them after the fact.

Conclusion

Each incident is an expensive lesson; this commission settlement anomaly was contained within twenty minutes of stop‑loss and nine hours of rollback, keeping actual financial loss minimal. More importantly, the team built a perception system that detects and blocks issues before they grow, reinforcing that fast detection, rapid stop‑loss, and swift repair are the true metrics of operational resilience.