When Balance Rollback Triggers 8 Million in Duplicate Withdrawals – Lessons Learned

The biggest fear in payments is duplicate payouts.

Even a small loss is problematic, and a large loss can cause bankruptcy, so fund safety is critical.

I experienced this terrifying incident and share it so others can learn.

The incident was caused by an account balance rollback.

The wallet shows a balance based on the underlying account balance, allowing users to withdraw.

To support business growth, we needed a global merchant ID (C‑ID), replacing the separate IDs used by each business line, a process we call “data sweeping”.

However, the system had an aggressive exception handling mechanism called “system rollback”, which included balance rollbacks.

We assumed no users would request withdrawals at night, but that assumption proved wrong.

Our merchants are night owls; within minutes, many initiated withdrawals, the balance rolled back, the withdrawal appeared to fail, the balance returned, and they tried again, causing duplicate withdrawals.

The newly added balance was created by the rollback and did not represent actual income, making the account inconsistent.

Account balance ≠ sum of total transaction flow.

Because the incident lasted only a short time, the fund‑safety mechanisms did not detect it, resulting in about 8 million of duplicate withdrawals within ten minutes.

The next day we quickly identified inconsistent accounts, generated an “abnormal withdrawal recovery list”, and sent it to operations to reclaim the funds from merchants.

To prevent merchants’ future earnings from being withdrawn and to make accounts consistent, we inserted a “‑duplicate withdrawal” adjustment, turning the merchants’ balances negative; subsequent income automatically repaid this “debt”.

Most merchants, understanding the situation, returned the excess money to the company’s public account.

After the system received the repayments, we recorded a “duplicate withdrawal reversal” transaction, bringing the merchants’ wallet balances from negative to zero.

Learning from this mistake, we added three safeguards: stricter balance control, self‑consistency checks for outgoing accounts, and regular reconciliation with the general ledger.

Fund‑security golden rule: record only after the channel succeeds, and debit only after the deduction succeeds.