Designing a Billion‑User Payment System: Final Architecture Review

From Requirement to Architecture: Full Payment Request Flow

A 150‑yuan composite payment order illustrates the core design across fourteen articles: the user orders 150 yuan, selects balance payment, and the backend splits the amount into 30 yuan free‑card, 100 yuan wallet, and 20 yuan WeChat.

Gateway layer (Article 6) – The request first reaches service‑pay‑gateway, which validates parameters, verifies signatures, and normalises request formats from APP, Mini‑Program, etc. It also enriches channel information such as retrieving the user’s OpenID from the user service.

Orchestration layer (Article 7) – The request is handed to payment‑orchestration. It recognises a CENTER‑type composite payment, invokes the fee‑allocation engine to compute channel amounts (free‑card 30, wallet 100, WeChat 20), and sequentially initiates pre‑order calls. Internal channels are deducted synchronously; the third‑party WeChat channel obtains a prepayId. After all channels succeed, the client confirms payment, the WeChat callback arrives, and the results are merged and sent to the order system via RocketMQ.

Channel layer (Articles 8‑10) – Each payment channel is an independent microservice (e.g., wechat‑base, alipay‑base, douyin‑base, hk‑base, jp‑base, mo‑base). These services encapsulate the SDKs, signatures, and HTTP calls of their respective providers. The orchestration layer accesses them through a Dubbo client anti‑corruption layer, never depending on a specific SDK.

Data layer (Article 4) – Payment orders use a two‑level structure: a master record for overall status and detail records for each channel’s transaction ID and status. A strict state machine enforces the flow NEW → PROCESSING → SUCCESS/FAIL, prohibiting backward transitions.

MQ notification (Article 12) – Once all channel callbacks succeed, the orchestration layer publishes a message to the order system. The message contains the master order result and a nested list of channel results; the order system updates its status and triggers fulfilment.

Refund (Article 11) – A partial refund of 50 yuan follows a template method that respects channel priority: WeChat 20 yuan (asynchronous third‑party refund) first, then wallet 30 yuan (synchronous internal refund). Refund and payment state machines operate independently.

Reconciliation (Articles 13‑14) – At 02:00 am daily, two reconciliation jobs run. Internal reconciliation compares order‑system and payment‑system states for consistency; channel reconciliation compares payment records with third‑party statements for amount consistency. Discrepancies are posted to a DingTalk group for manual verification.

Why the Three‑Layer Design Works

Gateway layer isolates terminal differences (APP, WeChat Mini‑Program, Alipay Mini‑Program, Douyin Mini‑Program). Adding a new terminal only changes the gateway.

Orchestration layer coordinates fee allocation, transaction coordination, and callback merging without knowing channel‑specific details. It uses a unified PaymentChannelProcessor interface; adding a new channel requires only implementing this interface.

Base layer provides physical isolation of channels. Updating the WeChat SDK touches only wechat‑base; other channels remain unaffected. A failure in one channel does not affect the others.

Channel Independence at Billion‑User Scale

With 500 k daily orders, a single channel outage can affect hundreds of thousands of transactions. For example, a 2‑hour WeChat outage could impact over 100 k orders.

Independent microservices ensure that SDK upgrades or API changes for one provider are low‑risk for the rest. If Alipay experiences a network issue, WeChat and wallet payments continue unaffected. The orchestration layer can automatically downgrade or prompt the user to choose an alternative channel.

Adding a new channel (e.g., Douyin) required only creating a douyin‑base service that implements the three required interfaces; the orchestration code remained unchanged.

Independent deployment allows scaling each channel according to its traffic; the WeChat service can be enlarged without expanding the entire payment system.

Business‑Driven Architectural Choices

Design decisions are tied to concrete business conditions:

Daily 500 k orders fit in memory, so reconciliation uses two HashSet structures (O(n) diff) instead of a distributed engine like Spark; a larger volume would require a different approach.

Store closing at 22:00 enables a T+1 reconciliation window where data is stable; 24/7 businesses would need a different strategy.

Domestic‑focused business leads to three domestic channel services and three international ones; a purely domestic or international focus would adjust the service granularity.

Composite payments necessitate a full orchestration layer; single‑channel payments could simplify or eliminate this layer.

Reconciliation: The Final Safety Net

Internal reconciliation ensures system‑to‑system state consistency, while channel reconciliation guarantees monetary consistency. Differences may arise from delayed settlement, fee calculations, or test transactions, but each must be verified to maintain a complete audit trail.

Conclusion

The essence of the payment system is not the number of integrated channels but the ability to keep each channel isolated, prevent fault propagation, and contain change impact. The three‑layer architecture—gateway, orchestration, base—achieves this by limiting the scope of failures, enabling independent scaling, and simplifying future extensions.

When applying these ideas, understand your own business volume, operating hours, and geographic mix; then tailor the architecture accordingly rather than copying the exact design.