Designing a 12‑State Payment System State Machine from Pending to Completed

Problem Statement

The core issue in a payment system is not a slow callback but the lack of a clearly defined state transition model. When a payment callback arrives after an order has been cancelled, naive implementations can cause the order state to jump incorrectly, leading to inconsistent order fulfillment and financial mismatches.

Typical Concurrency Scenario

User creates an order (state CREATED).

User is redirected to a third‑party payment channel.

Network jitter or retry delays cause the payment callback to arrive tens of seconds later.

A timeout task may already have moved the order to CANCELLED or EXPIRED.

The delayed PAY_SUCCESS callback then tries to set the order state to SUCCESS.

If the system simply executes UPDATE t_order SET state='SUCCESS' WHERE id=?, the final state depends on which thread writes last, producing a classic check‑then‑act race.

Key Requirements

Orders must not jump to illegal states.

All state changes must be idempotent and auditable.

Concurrent updates must be safely rejected.

When a cancelled order receives a successful payment, the system should automatically start a refund flow instead of reverting to SUCCESS.

Real‑World Business Scenario (Flash Sale)

During a large promotion the following characteristics appear:

High order creation peaks.

Payment callbacks are delayed and may be duplicated.

Timeout jobs run in minute‑level batches.

Users may cancel while payment is in progress.

Payment channels may send out‑of‑order or duplicate callbacks.

Why a Separate Order and Payment State Machine?

Order state reflects fulfillment (e.g., FINISHED), while payment state reflects the financial side (e.g., SUCCESS). Mixing them leads to ambiguous situations such as an order being CANCELLED while the payment is already SUCCESS.

12 Defined Order States

CREATED

– order just created, awaiting payment. PAYING – payment request sent, waiting for result. SUCCESS – payment succeeded, fulfillment pending. FAILED – payment failed, can retry. CANCELLED – user or system cancelled, may already have deducted funds. EXPIRED – timeout closed, distinct from manual cancel. REFUNDING – refund in progress. REFUNDED – full refund completed (terminal). PARTIAL_REFUNDED – partial refund, may continue. FINISHED – order fulfilled, may still have after‑sale refunds. CLOSED – final closed state, no further flow. ABNORMAL – abnormal state requiring manual or compensating handling.

State Machine Axes

Order state machine (core focus of this article).

Payment state machine (tracks channel result).

Refund state machine (handles reverse cash flow).

Transition Table (Excerpt)

OrderTransition(CREATED, SUBMIT_PAY, PAYING)
OrderTransition(CREATED, USER_CANCEL, CANCELLED)
OrderTransition(CREATED, TIMEOUT_CLOSE, EXPIRED)
OrderTransition(PAYING, PAY_SUCCESS, SUCCESS)
OrderTransition(PAYING, PAY_FAIL, FAILED)
OrderTransition(PAYING, USER_CANCEL, CANCELLED)
OrderTransition(PAYING, TIMEOUT_CLOSE, EXPIRED)
OrderTransition(FAILED, SUBMIT_PAY, PAYING)
OrderTransition(FAILED, USER_CANCEL, CANCELLED)
OrderTransition(FAILED, TIMEOUT_CLOSE, EXPIRED)
OrderTransition(SUCCESS, COMPLETE, FINISHED)
OrderTransition(SUCCESS, REFUND_APPLY, REFUNDING)
OrderTransition(SUCCESS, MARK_ABNORMAL, ABNORMAL)
OrderTransition(CANCELLED, PAY_SUCCESS, REFUNDING)
OrderTransition(EXPIRED, PAY_SUCCESS, REFUNDING)
OrderTransition(CANCELLED, FORCE_CLOSE, CLOSED)
OrderTransition(EXPIRED, FORCE_CLOSE, CLOSED)
OrderTransition(REFUNDING, REFUND_SUCCESS, REFUNDED)
OrderTransition(REFUNDING, PARTIAL_REFUND_SUCCESS, PARTIAL_REFUNDED)
OrderTransition(REFUNDING, REFUND_FAIL, ABNORMAL)
OrderTransition(PARTIAL_REFUNDED, REFUND_APPLY, REFUNDING)
OrderTransition(PARTIAL_REFUNDED, COMPLETE, FINISHED)
OrderTransition(FINISHED, REFUND_APPLY, REFUNDING)
OrderTransition(ABNORMAL, REFUND_APPLY, REFUNDING)
OrderTransition(ABNORMAL, COMPLETE, FINISHED)
OrderTransition(REFUNDED, FORCE_CLOSE, CLOSED)

Why Not Simple If‑Else

Using if (order.getState() == PAYING) { … } works only in single‑threaded environments. In production, concurrent threads can read the same state and overwrite each other, leading to lost updates.

Correct Model: Transition Table + Conditional Update

The state machine stores a map of (currentState, event) → nextState. When an event occurs, the service loads the order, looks up the allowed transition, and performs a CAS update:

UPDATE t_order
SET state = ?, version = version + 1, update_time = NOW()
WHERE id = ? AND state = ? AND version = ?;

If the update count is zero, the state has been changed concurrently and a ConcurrentStateChangeException is thrown.

Event‑Driven Architecture (Outbox + Kafka)

All state changes are persisted together with an outbox record. A separate publisher reads NEW outbox rows, publishes them to Kafka, and marks them PUBLISHED. This decouples the callback transaction from downstream side‑effects such as order fulfillment, coupon issuance, or inventory deduction.

Idempotency Strategy

Two‑layer idempotency:

Fast Redis key ( pay:callback:{businessKey}) filters most duplicates.

MySQL unique constraint on ( channel_code, channel_trade_no) guarantees eventual consistency.

INSERT INTO t_pay_callback_record (channel_code, channel_trade_no, payload_json, trace_id)
VALUES (?,?,?,?)
ON DUPLICATE KEY UPDATE ...;

Callback Processing Flow

Generate idempotency key from channel code and trade number.

Attempt Redis consume; if already consumed, log and return.

Verify signature.

Load payment record, validate amount.

CAS update payment status from PAYING to SUCCESS.

Write PaySuccessEvent to outbox.

If the CAS update fails, the callback is considered already processed.

Refund Application Service

When a REFUND_APPLY event is received, the order state machine fires the event; if the resulting state is REFUNDING, the refund command service initiates automatic refund.

Kafka Consumer Example

@KafkaListener(topics = "pay-success-topic", groupId = "order-state-machine-group")
public void onMessage(PaySuccessEvent event) {
    OrderState target = orderStateMachine.fire(event.orderId(), OrderEvent.PAY_SUCCESS, "pay-success-consumer", event.traceId());
    if (target == OrderState.REFUNDING) {
        refundCommandService.applyAutoRefund(event.orderId(), event.paymentNo(), event.traceId());
        log.warn("order already closed, auto refund started, orderId={}, paymentNo={}, traceId={}", event.orderId(), event.paymentNo(), event.traceId());
    }
}

This demonstrates that the same PAY_SUCCESS event can lead to SUCCESS or REFUNDING depending on the current order state.

High‑Concurrency Engineering

Gateway‑level rate limiting for callbacks.

Callback endpoint performs only verification, idempotency, persistence, and event outbox; all heavy side‑effects are asynchronous.

No distributed lock by default; rely on DB CAS, version, and ordered Kafka consumption.

Separate thread pools for web, MQ consumption, external channel calls, and scheduled tasks to avoid resource contention.

Hot‑order handling: Redis fast idempotency, DB unique key fallback, exponential back‑off for retries, and conversion to ABNORMAL when automatic recovery fails.

Abnormal State Handling

The ABNORMAL state captures situations that cannot be automatically resolved (e.g., refund failure, amount mismatch). It provides a clear exit for monitoring, ticketing, and manual compensation.

Reconciliation & Compensation

Because callbacks may be lost or duplicated, a periodic reconciliation job pulls channel statements, compares them with local payment records, and performs corrective actions:

Missing successful payment → re‑emit PaySuccessEvent.

Local refund in progress but channel shows success → emit RefundSuccessEvent.

Unresolvable discrepancy → mark order ABNORMAL for manual handling.

Observability

Structured logs include

traceId, orderId, paymentNo, channelTradeNo, fromState, toState, event, operator

.

Metrics: callback volume, duplicate callbacks, signature failures, illegal transition count, CANCELLED/EXPIRED → REFUNDING count, auto‑refund success rate, ABNORMAL order count, Outbox backlog, reconciliation diff count.

Trace propagation across gateway, callback service, Kafka message key, refund service.

Alerting on spikes of ABNORMAL, Outbox backlog, refund failures, unreconciled payments.

Security & Risk Controls

Signature verification, timestamp, merchant ID, appId checks.

Replay protection via time windows and unique notify_id.

Amount, currency, order number, and channel trade number validation.

Sensitive data masking in logs.

Technology Choices

Custom lightweight state machine (readable, tightly coupled with DB CAS).

Kafka (or RocketMQ/RabbitMQ) for reliable asynchronous event delivery.

MySQL with version column and unique constraints.

Redis for fast idempotency cache.

Evolution Roadmap

Stage 1 – Single‑service prototype with state machine, conditional updates, and callback idempotency.

Stage 2 – Add Outbox and MQ to decouple side‑effects.

Stage 3 – Introduce reconciliation, abnormal‑state governance, monitoring, and multi‑environment rollout.

Implementation Checklist

Centralized transition table and validation.

All updates use WHERE state=? AND version=? CAS.

Unique key on ( channel_code, channel_trade_no) for final idempotency.

Outbox table present and publisher scheduled.

Redis fast idempotency layer with DB fallback.

Kafka consumer partitions by orderId to guarantee order‑wise processing.

Metrics and alerts for illegal transitions, ABNORMAL orders, Outbox backlog, refund failures.

Conclusion

The most dangerous situation in a payment system is not a failed request but a state being incorrectly advanced while the system believes it succeeded. By defining explicit state transitions, enforcing them with conditional updates, guaranteeing idempotency, and driving all side‑effects through an outbox‑MQ pipeline, the design ensures that "payment callback collides with order cancellation" is handled safely, automatically refunds when needed, and leaves a clear audit trail for any remaining anomalies.