Building a Payment System for Hundreds of Millions of Users – Part 1 Intro

One night a colleague reported that several orders showed as paid in the order system but no corresponding payment records existed; investigation uncovered a malicious actor who forged the free‑card callback format, directly invoking the order‑system callback API and causing false‑positive payment status.

The breach highlighted two deeper problems: the lack of a daily cross‑check between the order and payment systems, and insufficient security safeguards for internal payment channels. The team responded by adding two reconciliation mechanisms—one that aligns platform records with external channels (WeChat, Alipay, etc.) and another that compares each order’s recorded payment flow ID against the payment system daily, raising alerts for any mismatch.

The column series, authored by the former technical lead, documents the complete redesign of this payment system, which now supports over 100 million registered users and processes an average of 500 k orders per day. It covers the full payment lifecycle across 20 articles, from product requirements to code implementation and operational practices.

Module 1: Requirements & Solution Design (3 articles)

It starts with a PRD that enumerates all payment scenarios—APP, mini‑program, POS, virtual vs. physical goods, and combination payments involving free‑card, wallet balance, and WeChat/Alipay. It also defines refund cases (full, partial, batch) and multi‑merchant management, ensuring that design decisions are driven by real business pressures.

Module 2: Infrastructure (2 articles)

The data model uses a two‑level structure (master flow + detail records) to guarantee atomicity for composite payments. A state machine defines statuses (NEW, PROCESSING, SUCCESS, FAILURE, REVERSE) and transition rules. Configuration is organized in three layers (application, merchant, store) with tables payment_merchant, payment_application, and payment_merchant_store, enabling correct channel routing based on request source.

Module 3: Core Payment Flow (12 articles)

Key topics include:

Payment gateway vs. basic gateway—parameter enrichment and channel enrichment.

Orchestration and channel routing using the Strategy pattern; fee‑splitting logic (free‑card → wallet → third‑party).

Domestic channel integration (WeChat, Alipay) with MD5 vs. RSA2 signatures and two‑phase order flow.

Wallet redesign from outsourced to in‑house, handling dual accounting systems.

Free‑card flow that locks inventory, freezes amount, and performs automatic reversal on failure.

International channels with RSA‑SHA256 or HMAC‑SHA256 signatures, multi‑currency handling, and custom HTTP headers.

POS line‑of‑sight payments using an authorization‑code stored in Redis (30 s TTL) and front‑end polling for status TS03.

Combination payment implementation—atomic master‑detail writes, priority rules, and Redis‑Hash based callback aggregation.

Refund subsystem with a template‑method pattern, distributed lock via Redis, and priority‑based channel selection.

Callback idempotency across three layers (free‑card, account transaction, combination) and retry/compensation strategies.

Risk control integrating two third‑party services (Shumei, Tencent Cloud), blacklist caching, and tiered user experience based on risk level.

Module 4: Reconciliation (2 articles)

One article describes a factory‑pattern BillReader that normalizes CSV or text statements from WeChat and Alipay into a unified format, then classifies differences (missing on platform, missing on channel, amount mismatch). The second article details the order‑payment reconciliation process ( chargeBill, chargeOrder, chargeDiff) that narrows the detection window from days to hours.

Technical Stack

Language: Java 17

Framework: Spring Boot 2.7

Microservice framework: Spring Cloud Alibaba 2021.0.1.0

RPC: Dubbo 3.2

Service registry & config: Nacos (bundled with Spring Cloud Alibaba)

Rate‑limit & circuit‑break: Sentinel

ORM: MyBatis‑Plus 3.5

Sharding: ShardingSphere 5.3

Database: MySQL 8.0

Cache & distributed lock: Redis + Redisson 3.17

Message queue: RocketMQ (Spring Boot Starter 2.2)

Scheduler: XXL‑Job 2.4

Object mapping: MapStruct 1.5

Local cache: Caffeine 2.9

The payment system serves as a decision‑making training ground: with seven or eight channels, each with its own signing, callback format, and refund rules, engineers must constantly trade off performance, consistency, and user experience, and there is no one‑size‑fits‑all solution.