How to Design Distributed Transactions for Consistent Microservices

Distributed transaction scenarios should be designed by breaking a large transaction into small atomic transactions plus asynchronous messaging, achieving eventual consistency without relying on heavyweight distributed transaction protocols.

What Is a Transaction

Transaction (Transaction) and Its ACID Properties

Atomicity : The transaction is an indivisible unit; all changes are applied or none are.

Consistency : Data must be in a valid state before and after the transaction, preserving all integrity rules.

Isolation : The system isolates concurrent transactions so intermediate states are invisible to others.

Durability : Once committed, changes survive system failures.

Typical Scenario: Bank Transfer

For example, Li Lei transfers 100 units to Han Meimei. After a successful transaction, Li Lei’s balance drops from 500 to 400 and Han Meimei’s rises from 200 to 300. The transaction must maintain consistency and be idempotent.

E‑Commerce Scenario: Traffic Recharge

The core flow for a traffic‑recharge service includes:

User selects a traffic package on the purchase page.

System checks inventory and creates a recharge order.

User chooses a payment method (e.g., UnionPay, Alipay, WeChat) and pays.

After successful payment, the traffic is credited in near real‑time.

Distributed Transaction

Local transactions run on a single physical node, while distributed (or flexible) transactions span multiple nodes or services, making traditional locking ineffective. Flexible transactions achieve eventual consistency by using compensation, asynchronous guarantees, or best‑effort notifications.

Alipay’s flexible transaction implementations include four patterns:

Two‑phase commit

Compensation

Asynchronous guarantee

Best‑effort notification

Microservice Decomposition for the Traffic Center

Implemented with Dubbo, the system is split into:

Product Service

Order Service

Inventory Service

Payment Service

Direct‑Recharge Service

Message Service

Other auxiliary services

Scenario 1: Inventory‑Order Consistency

Uses compensation + best‑effort notification because the operations are fast and do not cross data centers.

User places an order, inventory is decremented first.

Order service is invoked.

If the order succeeds, both transactions commit.

If the order fails, inventory rolls back; if rollback fails, a delayed message retries until success.

Scenario 2: Order‑Payment‑Recharge Consistency

Adopts asynchronous guarantee due to long, cross‑region workflow and tolerable latency (1‑5 minutes).

Order service creates the order and sends a payment request; order status is “pending payment”.

User completes payment; the payment gateway notifies the traffic center, which marks the order as “paid”.

After payment, the traffic center enqueues a direct‑recharge task via the message service to decouple the long‑running CRM call.

If direct‑recharge succeeds, order status becomes “completed”.

If direct‑recharge fails, the order moves to “refund pending” and a refund is processed.

If direct‑recharge times out, a retry mechanism attempts up to a configured limit; unresolved orders are settled by a T+1 reconciliation process.

Scenario 3: Recharge Notification

After successful recharge, a best‑effort notification (push or SMS) is sent; failures trigger scheduled retries.

Scenario 4: Reconciliation and Settlement

Daily T+1 reconciliation compares payment records, traffic‑center orders, and CRM recharge logs to finalize order statuses. Settlement then matches payment gateway totals with financial records and provides detailed statements to provincial partners.

Traffic Center Architecture Design

The design follows microservice principles: each service is isolated and stateless where possible, with separate layers for request handling, computation, and storage. Data stores are vertically and horizontally partitioned (MySQL with master‑slave, Redis clusters, sharding via Mycat, Snowflake ID generation, etc.).

Performance scaling: add stateless compute nodes; monitor with Dubbo Monitor and Hystrix for dynamic scaling.

Capacity scaling: horizontal sharding of stateful storage using NoSQL (Codis) and relational databases (Mycat).

Storage optimization: use Redis/Tedis for read‑heavy workloads, Codis for balanced read/write, and MySQL master‑slave or multi‑master setups depending on data characteristics.

Never Give Up!