Three Crucial Architecture Decisions for Building a Reliable Payment System

Stability is a P0 requirement for payment systems, so the author first emphasizes that any architectural decision must prioritize fault isolation.

Decision 1 – Channel Isolation : The system integrates multiple channels such as WeChat, Alipay, Douyin, wallet, and free‑card. Because each channel has different stability, latency, and rate‑limiting characteristics, the author recommends deploying each channel in its own microservice rather than sharing a single service. If all channels share one service, a surge of slow requests from a failing channel can exhaust threads and affect every channel. The author cites an example where a WeChat outage caused the checkout UI to gray out WeChat, forcing users to switch to Alipay, demonstrating that isolated services allow selective shutdown without broader impact. The trade‑off is an increased number of services, which is justified by the reduced risk of cascading failures.

Decision 2 – Payment Gateway : Early in the project, the front‑end called the orchestration layer directly, which worked while the number of terminals was small. As the number of terminals grew (WeChat, native app, mini‑program, H5), request formats diverged. Packing all terminal‑specific logic (parameter validation, terminal identification, openId retrieval, request format conversion) into the orchestration layer made it increasingly heavyweight. The solution was to move this logic into a dedicated payment gateway, leaving the orchestration layer to handle only the payment flow itself. This separation keeps responsibilities clear and prevents the orchestration layer from becoming a bottleneck.

Decision 3 – Payment Callback Handling : Payment channels send asynchronous callbacks, often with retries, leading to possible duplicate processing. If two threads simultaneously check that all channels have completed callbacks, they may each publish a message to MQ, causing duplicate downstream work. The author’s approach stores each channel’s callback status in a RedisHash keyed by the payment order, then acquires a distributed lock per order before processing. This ensures that, regardless of how many retries a channel performs, only one thread advances the payment status, guaranteeing idempotent handling.

Collectively, these three designs address different failure modes—channel isolation prevents cross‑channel contagion, the gateway abstracts terminal differences, and the callback mechanism ensures consistency. The overarching goal is to stop a local fault from escalating into a system‑wide outage, a benefit that becomes evident only when a channel experiences real‑world anomalies.