Scaling Payment Systems: Sharding, Snowflake IDs, and High‑Availability

Database Sharding

In the era of Redis and Memcached, supporting 100,000 reads per second is straightforward, but processing the same number of writes for a payment system requires sharding the order table across multiple databases and tables. The strategy uses the user ID (uid) to determine both the database and table.

We adopt a binary‑tree expansion model, scaling databases by powers of two, which simplifies data synchronization during expansion. Each database contains ten order tables (order_0 … order_9), resulting in eight databases (DB1‑DB8) each with ten tables.

Database number = (uid / 10) % 8 + 1 Table number = uid % 10

For example, uid = 9527 maps to DB1 and table order_7.

Two types of sharding tools exist: client‑side sharding (direct DB connection) and middleware sharding. The client‑side approach offers 15‑20% higher performance, while middleware provides clearer module separation and easier DBA management. The authors chose a client‑side solution using their open‑source framework “Mango”.

Order ID Generation

Generating globally unique order IDs at 100,000 per second cannot rely on database auto‑increment IDs. Instead, a Snowflake‑style algorithm is used, combining a millisecond timestamp, a machine identifier, and a per‑millisecond sequence.

To enable direct lookup of the sharding location from the order ID, the ID also encodes the database and table numbers. The basic sharding info is a two‑character string: first digit for database (1‑8) and second for table (0‑9). To future‑proof scaling to up to 64 databases, the database part is expanded to two digits using modulo 64, resulting in a three‑character sharding suffix.

Eventual Consistency

To support queries by both user ID and business ID (bid), the order data is duplicated across two clusters: one sharded by uid and another by bid. Synchronous strong consistency would degrade performance, so asynchronous consistency is achieved using a message queue for data replication and a real‑time monitoring service to detect and reconcile differences.

Database High Availability

High availability ensures that database failures are quickly recovered without service disruption. A classic primary‑replica setup is described, where web servers connect to a virtual IP that points to the primary. If the primary fails, a backup replica is promoted automatically, and the virtual IP is remapped, restoring service within seconds.

Load balancers (LVS) are used for read replicas, allowing automatic failover without manual configuration changes.

Data Tiering

Data is divided into three tiers: (1) core order and transaction data stored only in the database for strict consistency, (2) user‑related data cached in Redis, and (3) configuration data cached in local memory. A high‑availability message push platform synchronizes configuration changes across servers to keep the local cache consistent.

Coarse‑Fine Traffic Control

To protect the system from traffic spikes, a coarse‑fine pipeline is placed before the web cluster. The coarse gate caps incoming requests at 1 million per second, discarding excess traffic, while the fine gate limits traffic to the web cluster to 100,000 requests per second, queuing the remainder for later processing.