SpringBoot + WeChat Pay V3: Architecture for Million-Concurrent Payments

Why payment chains fail during promotions

Typical integration steps are: user clicks pay → server calls WeChat order API → QR code or cashier is shown → WeChat sends a success callback → merchant updates order status. In production the chain faces four pressures:

Traffic pressure : a burst of order requests stresses the order‑creation interface, inventory lock, order DB, connection pool and thread pool.

Consistency pressure : user payment, order settlement, inventory deduction and coupon redemption must eventually be consistent.

External‑dependency pressure : network jitter, slow responses, certificate anomalies or delayed callbacks from WeChat can drag down the local pipeline.

Idempotency pressure : duplicate callbacks, repeated front‑end submissions or retry queries require a unified idempotent model, otherwise order states become chaotic.

Operational pressure : certificate rotation, hot‑config updates, payment‑delay alerts, callback retries, manual compensation and reconciliation are daily ops tasks.

Most incidents are blamed on “unstable WeChat interfaces”, but the root cause is usually that the merchant system does not treat the payment flow as an independent, governable transaction pipeline.

Core questions addressed

What changes does WeChat Pay V3 introduce compared with V2, and why does its security model affect system architecture?

Why can a payment chain not rely solely on a synchronous callback to update an order?

How should a high‑concurrency payment state machine, asynchronous decoupling, idempotency, compensation and reconciliation be designed?

How to turn WeChat Pay V3 into a production‑grade component in a SpringBoot project rather than a simple code snippet?

Real goal of a payment system: proven fund state

Payment is a multi‑step state transition across systems:

Place Order → Create Merchant Order → Call WeChat Order API → User Paying → WeChat Success Notification → Verify Signature & Decrypt → Inbound Event → Order State Transition → Post‑processing (inventory, coupons, points, fulfillment) → Reconciliation & Compensation Confirmation

Three guarantees are required:

User's money must not be misrecorded

Order state must not jump arbitrarily

After a fault the system must be able to compensate the account

Consequently the core objectives become:

Correctness over throughput

Final consistency over immediate synchronous responses

Auditability and compensation over one‑off lucky successes

If these goals are not met, adding caches, MQ, K8s or service meshes only amplifies complexity.

WeChat Pay V3 security model and its impact

V3 is a security model upgrade

Four key changes compared with the old version:

JSON payloads replace XML.

Request signatures use RSA instead of simple MD5/HMAC concatenation.

Callback notifications must be verified and then decrypt sensitive fields.

Platform certificates must be rotated periodically; merchants cannot assume certificates are immutable.

This turns integration into a channel layer with a security boundary:

Request side : merchant signs requests with its private key.

Response/Callback side : merchant verifies the signature using WeChat's platform certificate.

Notification data side : sensitive content is decrypted with the APIv3 Key.

Two production‑level risks arise if this model is treated as a black‑box SDK:

Signature or verification failures are fund‑chain risk events , not ordinary business exceptions.

Failure to rotate platform certificates puts the whole payment system into an unverifiable state .

Three key credentials

Merchant Private Key : used for request signing. Failure → WeChat rejects request; order/lookup fails.

WeChat Platform Certificate / Public Key : used to verify WeChat responses and callbacks. Failure → cannot confirm authenticity; fund risk.

APIv3 Key : used to decrypt notification resource data. Failure → cannot read transaction plaintext.

Why “update order immediately on callback” is dangerous

Callback is an external event, not a transaction boundary. Performing the full business flow inside the callback thread can cause:

WeChat thinks the notification was received, but the local business only completed half of the steps.

Callback thread blocks too long, causing WeChat retries and duplicate consumption.

During callback spikes the database becomes a bottleneck, dragging down the entire payment entry point.

The callback thread should only perform five actions:

Extract callback headers and raw payload.

Verify signature.

Decrypt.

Record inbound event and idempotent log.

Publish an internal payment‑success event.

Production‑grade payment architecture

Recommended high‑concurrency architecture

User / App / Mini‑Program → API Gateway → Order Service → Payment Orchestration Service → Channel Adapter (WeChat Pay V3 / future Alipay) → Redis Idempotency & Short‑Term State Cache → MySQL Payment Order & Event Tables → MQ Event Bus → Order Domain Consumer → Inventory / Coupon / Points / Fulfillment Consumers → Reconciliation & Compensation Tasks → Observability & Alerting System

Four layers are identified:

Access Layer:   order creation API, payment creation API, payment query API, callback endpoint
Execution Layer: WeChat client, signature verification, order query, close order, refund
State Layer:    payment order, payment event, idempotent log, compensation task, reconciliation result
Governance Layer: rate limiting, circuit breaking, timeout, certificate rotation, monitoring, alerting, audit, manual compensation

All four layers are indispensable. Many systems only implement the execution layer, leaving state and governance missing, which forces post‑mortem debugging via log hunting.

Why payment service must be decoupled from order service

Payment Service cares about channel interaction, signature verification, payment‑order state and fund event trustworthiness.

Order Service cares about order status, inventory confirmation and marketing redemption.

Hard coupling leads to model pollution, excessive domain dependencies, difficult compensation evolution and channel‑specific logic leaking into the order service.

Correct approach:

Payment service emits a unified payment domain event.

Order service consumes the event with business semantics.

Order updates must be idempotent.

Dedicated "payment order" tables

At least three core tables are recommended: t_order – business order. t_pay_order – payment order (multiple attempts per business order are possible). t_pay_event – payment notifications/events.

Reasons:

An order may have multiple payment attempts.

Payment state and order state are not one‑to‑one.

Original callback, verification result and channel transaction IDs need independent audit trails.

Reconciliation, compensation, refund and error handling require traceable payment‑level data.

Payment state machine

State definitions

INIT → WAITING_USER_PAY → PAYING → PAY_SUCCESS → PAY_FAILED → CLOSED → REFUND_PROCESSING → REFUNDED

Legal transitions (examples):

INIT → WAITING_USER_PAY (payment order created)

WAITING_USER_PAY → PAY_SUCCESS (callback or query confirms success)

WAITING_USER_PAY → CLOSED (timeout or order cancellation)

PAY_SUCCESS → REFUND_PROCESSING (refund initiated)

REFUND_PROCESSING → REFUNDED (refund succeeded)

Illegal transitions such as CLOSED → PAY_SUCCESS must be rejected or routed to an error/manual queue.

Two‑stage confirmation

channel_trade_state = SUCCESS

: WeChat has deducted the money. biz_process_state = PROCESSED / PENDING / FAILED: Local system has recorded the order.

This separation makes it clear whether money was actually deducted and whether the local business has caught up.

Robust state‑machine principles

Unidirectional progression – states only move forward toward the final result.

Idempotent updates – repeated identical events must not change the result.

Conditional updates – use WHERE status = ? to enforce idempotency.

Event traceability – every state change leaves an event record.

Recoverable failures – failed transitions can be retried or handled manually.

Real‑world high‑concurrency scenario – flash‑sale order payment flow

Scenario:

Limited‑time flash‑sale starts at 20:00.

Peak order traffic 120,000 QPS; effective payment‑creation requests ~8,000 QPS.

WeChat Native is the primary channel; users must complete payment within 15 minutes.

Key design points:

Filter invalid requests before they reach the payment pipeline.

Decouple payment result handling from the synchronous flow.

Converge all uncertainties into the state machine and compensation processes.

Overall process:

Flash‑sale eligibility → Create business order (pending payment) → Create payment order (WAITING_USER_PAY) → Call WeChat to obtain code_url → Return QR to front‑end → User pays → WeChat callback enters payment service → Idempotent inbound + event persistence + publish PAY_SUCCESS event → Order service consumes and updates order status → Asynchronous inventory, coupon, points processing → Periodic query & reconciliation as final fallback

SpringBoot integration – production‑grade code

1. Maven dependencies

<dependencies>
    <dependency>
        <groupId>com.github.wechatpay-apiv3</groupId>
        <artifactId>wechatpay-java-core</artifactId>
        <version>${wechatpay.java.version}</version>
    </dependency>
    <dependency>
        <groupId>com.github.wechatpay-apiv3</groupId>
        <artifactId>wechatpay-java-service</artifactId>
        <version>${wechatpay.java.version}</version>
    </dependency>
    <dependency>
        <groupId>com.github.wechatpay-apiv3</groupId>
        <artifactId>wechatpay-java-apache-httpclient</artifactId>
        <version>${wechatpay.java.version}</version>
    </dependency>
</dependencies>

Two practical recommendations:

Centralise SDK initialisation in a configuration component instead of scattering it across business code.

Never hard‑code private keys or APIv3 keys in configuration files; store them in Secret/Vault/KMS for production.

2. Payment configuration object

package com.example.pay.config;

import jakarta.validation.constraints.NotBlank;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.validation.annotation.Validated;

@Data
@Validated
@ConfigurationProperties(prefix = "payment.wechat")
public class WechatPayProperties {
    @NotBlank private String appId;
    @NotBlank private String mchId;
    @NotBlank private String mchSerialNo;
    @NotBlank private String privateKeyPem;
    @NotBlank private String apiV3Key;
    @NotBlank private String notifyUrl;
    private int connectTimeoutMillis = 2000;
    private int readTimeoutMillis = 5000;
    private int maxConnTotal = 200;
    private int maxConnPerRoute = 100;
}

Explicit timeout and connection‑pool parameters are crucial; default values are often the root cause of incidents during spikes.

3. WeChat HTTP client configuration

package com.example.pay.config;

import com.github.wechatpay.apache.httpclient.WechatPayHttpClientBuilder;
import com.github.wechatpay.apache.httpclient.auth.AutoUpdateCertificatesVerifier;
import com.github.wechatpay.apache.httpclient.auth.PrivateKeySigner;
import com.github.wechatpay.apache.httpclient.auth.WechatPay2Credentials;
import java.io.ByteArrayInputStream;
import java.nio.charset.StandardCharsets;
import java.security.PrivateKey;
import lombok.extern.slf4j.Slf4j;
import org.apache.hc.client5.http.config.RequestConfig;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager;
import org.apache.hc.core5.util.Timeout;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Slf4j
@Configuration
public class WechatPayClientConfig {
    @Bean
    public CloseableHttpClient wechatPayHttpClient(WechatPayProperties properties) throws Exception {
        PrivateKey merchantPrivateKey = PemUtils.loadPrivateKey(
                new ByteArrayInputStream(properties.getPrivateKeyPem().getBytes(StandardCharsets.UTF_8)));
        AutoUpdateCertificatesVerifier verifier = new AutoUpdateCertificatesVerifier(
                new WechatPay2Credentials(properties.getMchId(), new PrivateKeySigner(properties.getMchSerialNo(), merchantPrivateKey)),
                properties.getApiV3Key().getBytes(StandardCharsets.UTF_8));
        PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
        connectionManager.setMaxTotal(properties.getMaxConnTotal());
        connectionManager.setDefaultMaxPerRoute(properties.getMaxConnPerRoute());
        RequestConfig requestConfig = RequestConfig.custom()
                .setConnectTimeout(Timeout.ofMilliseconds(properties.getConnectTimeoutMillis()))
                .setResponseTimeout(Timeout.ofMilliseconds(properties.getReadTimeoutMillis()))
                .build();
        log.info("init wechat pay http client, mchId={}, maxConnTotal={}, maxConnPerRoute={}",
                properties.getMchId(), properties.getMaxConnTotal(), properties.getMaxConnPerRoute());
        return WechatPayHttpClientBuilder.create()
                .withMerchant(properties.getMchId(), properties.getMchSerialNo(), merchantPrivateKey)
                .withValidator(response -> true)
                .withWechatPay(verifier)
                .setConnectionManager(connectionManager)
                .setDefaultRequestConfig(requestConfig)
                .build();
    }
}

Key engineering points:

Explicitly configure connection pool.

Explicitly configure request timeouts.

Integrate automatic certificate update into client initialisation.

Make initialisation logs observable.

4. Payment order creation – validate business first

package com.example.pay.application;

import com.example.pay.domain.PayOrder;
import com.example.pay.domain.PayOrderStatus;
import com.example.pay.domain.repository.PayOrderRepository;
import com.example.pay.facade.OrderFacade;
import com.example.pay.gateway.WechatNativePayGateway;
import com.example.pay.web.command.CreatePayCommand;
import java.time.LocalDateTime;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Service
@RequiredArgsConstructor
public class PayApplicationService {
    private final OrderFacade orderFacade;
    private final PayOrderRepository payOrderRepository;
    private final WechatNativePayGateway wechatNativePayGateway;

    @Transactional
    public CreatePayResult createNativePay(CreatePayCommand command) {
        // 1. Validate order eligibility
        OrderSnapshot order = orderFacade.requirePayableOrder(command.orderNo(), command.userId());
        validateAmount(order.getPayAmountFen(), command.amountFen());
        // 2. Reuse a still‑valid payment order for duplicate rapid requests
        PayOrder existing = payOrderRepository.findLatestActiveByOrderNo(command.orderNo()).orElse(null);
        if (existing != null && existing.getStatus() == PayOrderStatus.WAITING_USER_PAY) {
            return new CreatePayResult(existing.getPayOrderNo(), existing.getCodeUrl(), existing.getExpireTime());
        }
        // 3. Create a new payment order and persist it in the same transaction
        PayOrder payOrder = PayOrder.createWaiting(
                command.orderNo(), command.userId(), command.amountFen(), LocalDateTime.now().plusMinutes(15));
        payOrderRepository.save(payOrder);
        // 4. Call WeChat to obtain the QR code
        String codeUrl = wechatNativePayGateway.createOrder(
                payOrder.getPayOrderNo(), order.getTitle(), command.amountFen(), payOrder.getExpireTime());
        // 5. Write back the QR code URL
        payOrderRepository.updateCodeUrl(payOrder.getPayOrderNo(), codeUrl);
        return new CreatePayResult(payOrder.getPayOrderNo(), codeUrl, payOrder.getExpireTime());
    }

    private void validateAmount(Integer orderAmountFen, Integer requestAmountFen) {
        if (!orderAmountFen.equals(requestAmountFen)) {
            throw new IllegalArgumentException("payment amount mismatch");
        }
    }
}

Key points:

Validate order eligibility before calling WeChat.

Persist a separate payment order to keep a trace of each attempt.

Reuse a still‑valid payment order for rapid duplicate requests.

Wrap creation and local state change in a transaction; write the channel result back afterwards.

5. WeChat Native pay gateway – hide channel details

package com.example.pay.gateway;

import com.example.pay.config.WechatPayProperties;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.nio.charset.StandardCharsets;
import java.time.LocalDateTime;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import java.util.HashMap;
import java.util.Map;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.hc.client5.http.classic.methods.HttpPost;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
import org.apache.hc.core5.http.ContentType;
import org.apache.hc.core5.http.io.entity.EntityUtils;
import org.apache.hc.core5.http.io.entity.StringEntity;
import org.springframework.stereotype.Component;

@Slf4j
@Component
@RequiredArgsConstructor
public class WechatNativePayGateway {
    private static final String NATIVE_PAY_URL = "https://api.mch.weixin.qq.com/v3/pay/transactions/native";
    private final CloseableHttpClient wechatPayHttpClient;
    private final WechatPayProperties properties;
    private final ObjectMapper objectMapper;

    public String createOrder(String payOrderNo, String description, Integer amountFen, LocalDateTime expireTime) {
        try {
            Map<String, Object> body = new HashMap<>();
            body.put("appid", properties.getAppId());
            body.put("mchid", properties.getMchId());
            body.put("description", description);
            body.put("out_trade_no", payOrderNo);
            body.put("notify_url", properties.getNotifyUrl());
            body.put("time_expire", expireTime.atOffset(ZoneOffset.ofHours(8))
                    .format(DateTimeFormatter.ISO_OFFSET_DATE_TIME));
            Map<String, Object> amount = new HashMap<>();
            amount.put("total", amountFen);
            amount.put("currency", "CNY");
            body.put("amount", amount);
            HttpPost request = new HttpPost(NATIVE_PAY_URL);
            request.setHeader("Accept", "application/json");
            request.setEntity(new StringEntity(objectMapper.writeValueAsString(body), ContentType.APPLICATION_JSON, StandardCharsets.UTF_8));
            try (CloseableHttpResponse response = wechatPayHttpClient.execute(request)) {
                String responseBody = EntityUtils.toString(response.getEntity(), StandardCharsets.UTF_8);
                int code = response.getCode();
                if (code < 200 || code >= 300) {
                    log.error("wechat native pay failed, payOrderNo={}, code={}, body={}", payOrderNo, code, responseBody);
                    throw new PaymentChannelException("wechat native pay failed");
                }
                Map<String, Object> result = objectMapper.readValue(responseBody, Map.class);
                return (String) result.get("code_url");
            }
        } catch (Exception ex) {
            log.error("create wechat native order error, payOrderNo={}", payOrderNo, ex);
            throw new PaymentChannelException("create wechat native order error", ex);
        }
    }
}

Important notes:

Log business order numbers but never log sensitive credentials or full certificates.

Convert channel exceptions into a unified local exception model to keep business code clean.

6. Callback controller – trusted event ingress

package com.example.pay.web;

import com.example.pay.application.PayNotifyApplicationService;
import jakarta.servlet.http.HttpServletRequest;
import java.io.BufferedReader;
import java.util.stream.Collectors;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@Slf4j
@RestController
@RequestMapping("/api/payments/wechat/notify")
@RequiredArgsConstructor
public class WechatPayNotifyController {
    private final PayNotifyApplicationService payNotifyApplicationService;

    @PostMapping
    public ResponseEntity<WechatNotifyResponse> notify(HttpServletRequest request,
            @RequestHeader("Wechatpay-Serial") String serial,
            @RequestHeader("Wechatpay-Signature") String signature,
            @RequestHeader("Wechatpay-Timestamp") String timestamp,
            @RequestHeader("Wechatpay-Nonce") String nonce) {
        String rawBody = readBody(request);
        try {
            payNotifyApplicationService.handleNotify(serial, signature, timestamp, nonce, rawBody);
            return ResponseEntity.ok(new WechatNotifyResponse("SUCCESS", "成功"));
        } catch (InvalidSignatureException ex) {
            log.warn("wechat notify signature invalid, serial={}, timestamp={}", serial, timestamp);
            return ResponseEntity.status(HttpStatus.UNAUTHORIZED)
                    .body(new WechatNotifyResponse("FAIL", "signature invalid"));
        } catch (Exception ex) {
            log.error("wechat notify process error", ex);
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
                    .body(new WechatNotifyResponse("FAIL", "internal error"));
        }
    }

    private String readBody(HttpServletRequest request) {
        try (BufferedReader reader = request.getReader()) {
            return reader.lines().collect(Collectors.joining(System.lineSeparator()));
        } catch (Exception ex) {
            throw new IllegalStateException("read notify body error", ex);
        }
    }

    public record WechatNotifyResponse(String code, String message) {}
}

Responsibilities of the callback endpoint:

Extract headers and raw body.

Verify signature.

Decrypt.

Persist inbound event and idempotent log.

Publish an internal payment‑success event.

The endpoint never directly modifies the order; it only acts as a trustworthy event source.

7. Payment notify application service – verify, decrypt, persist, publish

package com.example.pay.application;

import com.example.pay.domain.PayEvent;
import com.example.pay.domain.repository.PayEventRepository;
import com.example.pay.domain.repository.PayOrderRepository;
import com.example.pay.messaging.PayEventPublisher;
import com.example.pay.service.WechatNotifyVerifier;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Slf4j
@Service
@RequiredArgsConstructor
public class PayNotifyApplicationService {
    private final WechatNotifyVerifier wechatNotifyVerifier;
    private final PayEventRepository payEventRepository;
    private final PayOrderRepository payOrderRepository;
    private final PayEventPublisher payEventPublisher;

    @Transactional
    public void handleNotify(String serial, String signature, String timestamp, String nonce, String rawBody) {
        var notifyResult = wechatNotifyVerifier.verifyAndDecrypt(serial, signature, timestamp, nonce, rawBody);
        String transactionId = notifyResult.transactionId();
        // Idempotent check
        if (payEventRepository.existsByTransactionIdAndEventType(transactionId, "WECHAT_PAY_NOTIFY")) {
            log.info("duplicate wechat notify ignored, transactionId={}", transactionId);
            return;
        }
        // Persist the raw event for audit and replay
        PayEvent payEvent = PayEvent.fromWechatNotify(
                notifyResult.outTradeNo(), transactionId, rawBody, notifyResult.plainText());
        payEventRepository.save(payEvent);
        // Update payment order state
        int updated = payOrderRepository.markChannelSuccess(
                notifyResult.outTradeNo(), transactionId, notifyResult.successTime());
        if (updated == 0) {
            log.warn("pay order state not updated, outTradeNo={}, transactionId={}", notifyResult.outTradeNo(), transactionId);
        }
        // Publish internal event after persistence
        payEventPublisher.publishPaySuccess(new PaySuccessEvent(
                notifyResult.outTradeNo(), transactionId, notifyResult.successTime()));
    }
}

The crucial ordering is: persist the payment event first, then publish the internal event. This guarantees that even if MQ publishing succeeds but local persistence fails (or vice‑versa), the system can recover.

8. Order‑side consumption – idempotent update

package com.example.order.messaging;

import com.example.order.application.OrderPaymentService;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Slf4j
@Component
@RequiredArgsConstructor
public class PaySuccessConsumer {
    private final OrderPaymentService orderPaymentService;

    @KafkaListener(topics = "pay.success", groupId = "order-payment-group")
    public void onMessage(String message) {
        PaySuccessEvent event = JsonUtils.parse(message, PaySuccessEvent.class);
        orderPaymentService.handlePaySuccess(event);
    }
}

package com.example.order.application;

import com.example.order.domain.repository.OrderRepository;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Slf4j
@Service
@RequiredArgsConstructor
public class OrderPaymentService {
    private final OrderRepository orderRepository;

    @Transactional
    public void handlePaySuccess(PaySuccessEvent event) {
        int updated = orderRepository.markPaidIfWaiting(event.orderNo(), event.transactionId(), event.successTime());
        if (updated == 1) {
            log.info("order marked paid, orderNo={}, transactionId={}", event.orderNo(), event.transactionId());
            return;
        }
        if (orderRepository.isPaid(event.orderNo())) {
            log.info("duplicate pay success event ignored, orderNo={}", event.orderNo());
            return;
        }
        log.warn("unexpected order state for pay success, orderNo={}, transactionId={}", event.orderNo(), event.transactionId());
        throw new IllegalStateException("unexpected order state");
    }
}

Idempotent handling relies on conditional updates such as WHERE order_status = 'WAIT_PAY' and falls back to state checks when the update count is zero.

Compensation system – catching lost callbacks and mid‑flow failures

Callback is not a guaranteed input; query and reconciliation are the ultimate decision mechanisms.

Compensation tasks should:

Scan payment orders that have been waiting too long.

Call WeChat query API to confirm channel status.

For channel‑success but local‑missing records, push a compensation event.

For timed‑out unpaid orders, attempt to close them.

Record compensation results and attempt counts.

Compensation scheduler

package com.example.pay.scheduler;

import com.example.pay.application.PayCompensationService;
import java.util.List;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@Slf4j
@Component
@RequiredArgsConstructor
public class PayCompensationScheduler {
    private final PayCompensationService payCompensationService;

    @Scheduled(fixedDelayString = "${payment.compensation.fixed-delay-ms:60000}")
    public void compensate() {
        List<String> payOrderNos = payCompensationService.findNeedCompensatePayOrders(200);
        for (String payOrderNo : payOrderNos) {
            try {
                payCompensationService.compensateSingle(payOrderNo);
            } catch (Exception ex) {
                log.error("compensate pay order error, payOrderNo={}", payOrderNo, ex);
            }
        }
    }
}

Compensation service logic

package com.example.pay.application;

import com.example.pay.domain.PayOrder;
import com.example.pay.domain.repository.PayOrderRepository;
import com.example.pay.gateway.WechatPayQueryGateway;
import com.example.pay.messaging.PayEventPublisher;
import java.util.List;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Service
@RequiredArgsConstructor
public class PayCompensationService {
    private final PayOrderRepository payOrderRepository;
    private final WechatPayQueryGateway wechatPayQueryGateway;
    private final PayEventPublisher payEventPublisher;

    public List<String> findNeedCompensatePayOrders(int limit) {
        return payOrderRepository.findTimeoutWaitingPayOrders(limit);
    }

    @Transactional
    public void compensateSingle(String payOrderNo) {
        PayOrder payOrder = payOrderRepository.requireByPayOrderNo(payOrderNo);
        WechatTradeState tradeState = wechatPayQueryGateway.query(payOrderNo);
        switch (tradeState) {
            case SUCCESS -> {
                int updated = payOrderRepository.markChannelSuccessIfWaiting(payOrderNo);
                if (updated == 1) {
                    payEventPublisher.publishPaySuccessFromCompensation(payOrder);
                }
            }
            case CLOSED, REVOKED, PAYERROR -> payOrderRepository.markClosedIfWaiting(payOrderNo);
            case NOTPAY, USERPAYING -> payOrderRepository.increaseCompensateCount(payOrderNo);
            default -> payOrderRepository.recordUnknownTradeState(payOrderNo, tradeState.name());
        }
    }
}

Compensation is not a naive periodic query; it respects state, audits each attempt, and applies limits to avoid creating a new traffic spike.

Idempotent compensation

Compensation tasks must also be idempotent and rate‑limited. Typical controls include:

Batch size per scan.

Maximum retry count per payment order.

Query API QPS limit.

Compensation task concurrency.

Back‑off strategy on repeated failures.

High‑concurrency governance beyond connection pools

Access layer – filter invalid traffic

User authentication.

Order ownership verification.

Order status check.

Amount consistency check.

Idempotency token validation.

Rate‑limiting and risk control.

Application layer – control synchronous resource consumption

HTTP connection pool.

Servlet container thread pool (Tomcat/Undertow/Netty).

Downstream DB connection pool.

MQ producer buffers.

Redis connection pool.

A typical failure chain during a spike:

Downstream WeChat slows → Application threads pile up → HTTP pool exhausted → DB connections reclaimed slowly → Callback processing blocked → WeChat retries → System avalanche

Mitigation requires timeout, isolation, rate‑limiting, back‑off and circuit‑breaker.

State layer – avoid hotspot writes

Shard by order or payment number.

Keep transaction scope short.

Avoid updating many business tables in a single transaction.

Asynchronously handle points, coupons, notifications.

Result‑query layer – protect backend from front‑end polling

Cache successful payment results in Redis for a short window.

Front‑end should use exponential back‑off instead of fixed‑interval polling.

Consider WebSocket/SSE push when feasible.

Query API returns only lightweight status.

Operational risks in boundary conditions

Duplicate notifications are normal

System must treat the first successful handling as final and quickly acknowledge subsequent duplicates.

Payment success vs. order closed conflict

If a user pays in the last second and the order timeout task closes the order, the incoming success callback creates a conflict. The correct handling is to mark the situation as "payment‑success‑but‑order‑closed" and route it to an error‑handling queue for manual or automated resolution (re‑open order, refund, or audit).

Certificate auto‑update is not set‑and‑forget

Startup fetch failures must be monitored.

Runtime update failures need alerts.

Updater thread exceptions must be observable.

Local cache corruption and recovery procedures are required.

Closing orders is not a simple cleanup

Before calling the close‑order API ensure:

Local order is already closed.

Channel side is unlikely to still be processing payment.

No window exists where payment could succeed after the close request.

Logging overhead can become a new bottleneck

Sample logs on the success path.

Full context logs on error paths.

Mask sensitive fields.

Archive raw callbacks in a structured store instead of flooding application logs.

Consistency design – event‑driven + Outbox + compensation

For a payment flow involving payment, order, inventory, coupon and points services, a strong synchronous distributed transaction is costly and risky.

Recommended pattern:

Payment service confirms the payment event within a local transaction.

Publish the payment‑success event via an Outbox table.

Downstream services consume the event idempotently.

Compensation and reconciliation provide eventual consistency.

Benefits: short main flow, retryable failures, independent idempotent downstream processing, auditability. Trade‑offs: need clear event model, accept eventual consistency, build compensation and reconciliation capabilities.

Observability – make accidents locate‑able, quantifiable, recoverable

Metrics

Payment order creation success rate.

P95/P99 latency of order creation.

WeChat callback receipt rate.

Signature verification failure count.

Inbound event backlog size.

MQ consumption latency.

Pending compensation order count.

Compensation query success rate.

Paid‑but‑not‑accounted‑order count.

Certificate update failure count.

Distributed tracing

Propagate identifiers across logs, traces and events: orderNo, payOrderNo, transactionId, requestId, traceId. This enables end‑to‑end tracing from user click to WeChat order creation, callback receipt, event persistence and order update.

Alerting

Five‑minute window of rising signature‑verification‑failure rate.

Compensation queue backlog exceeding threshold.

Payment‑success‑but‑order‑not‑accounted latency breach.

Certificate update failures.

Channel call error rate spikes.

Alerts should pinpoint the failing stage rather than fire a generic alarm.

Multi‑channel expansion – what to abstract

When adding Alipay, bank cards or aggregated payments, abstract only stable domain capabilities:

Create payment order.

Query order.

Close order.

Verify callback.

Initiate refund.

Query refund.

Define a simple channel interface:

public interface PayChannelGateway {
    CreateChannelPayResult createPay(ChannelCreatePayCommand command);
    ChannelTradeQueryResult queryPay(String payOrderNo);
    void closePay(String payOrderNo);
    ChannelNotifyResult verifyNotify(ChannelNotifyRequest request);
}

Do not force a one‑to‑one mapping of every channel‑specific field or status code; keep external differences encapsulated while exposing unified business‑level semantics.

Containerisation & deployment – stateless yet governed

Configuration & secret management

Non‑sensitive configs go to a config centre.

Private keys, APIv3 keys and certificates reside in Secret/Vault/KMS.

All config changes are audited.

Support gray‑release and rollback.

Elastic scaling

Ensure WeChat order‑creation rate limits align with local rate‑limiting.

MQ partition count must support consumer parallelism.

Database connection pool limits must be sufficient.

Redis must not have hotspot keys.

Otherwise scaling pods only amplifies downstream bottlenecks.

Release strategy

Use rolling or canary deployments.

Keep the callback endpoint highly available during releases.

Avoid change windows during major promotional periods.

Perform channel‑level smoke tests before full rollout.

If certificates or signing logic change, have a rollback plan.

Production checklist

Security & credentials

Merchant private key stored securely.

APIv3 key not in plain files.

Platform certificate auto‑update verified.

Certificate expiry monitored.

Core flow capabilities

Idempotent order creation interface.

Duplicate payment clicks reuse existing valid payment order.

Callback uses raw body for signature verification.

Callback only performs inbound convergence, not business updates.

Consistency & compensation

Payment order state machine explicitly modelled.

Order updates performed with conditional updates.

Payment event table or equivalent audit capability present.

Compensation query task exists.

Reconciliation mechanism in place.

High‑concurrency controls

HTTP connection pool explicitly configured.

Downstream timeouts explicitly configured.

Callback endpoint isolated with its own thread pool.

Payment result query uses cache or back‑off.

Ops & observability

Metrics for payment success rate, callback success rate, compensation backlog.

Signature‑failure and certificate‑update alerts.

Key identifiers flow through logs, traces and events.

Manual compensation and error‑handling procedures documented.

Scope & applicability

The design fits e‑commerce order payment, membership recharge, knowledge‑pay and high‑concurrency promotional payments. It does not directly address front‑end checkout complexity, cross‑border multi‑currency settlement, full accounting, or ultra‑high‑availability dual‑data‑center disaster recovery. Those require additional layers (fund accounts, profit‑sharing models, accounting vouchers, end‑of‑day reconciliation, etc.) built on top of the core payment pipeline described here.

Final takeaway

Integrating SpringBoot with WeChat Pay V3 is straightforward, but production‑grade reliability hinges on treating the payment flow as a state‑machine‑driven transaction pipeline, using the callback as a trusted event ingress, decoupling business side‑effects, establishing idempotent compensation, and building comprehensive observability. Only then does the system survive the inevitable uncertainties of traffic spikes, network jitter and component failures.