How to Solve Distributed Transactions, Idempotency, and Async Messaging in Microservices

Introduction

When splitting and upgrading legacy monoliths into microservices, developers often encounter distributed transaction problems, idempotency challenges, asynchronous message disorder, and compensation requirements. The following sections record practical solutions and thinking based on real‑world experience.

Distributed Transactions

During system decomposition, a typical scenario involves an order service calling a wallet service to deduct funds and update order status. Embedding a synchronous HTTP call inside a database transaction looks simple but introduces serious risks:

The wallet service may respond slowly, causing the database connection to remain open and exhaust the connection pool.

If the wallet service is down, the transaction rolls back, making the order service unavailable.

Network glitches can lead to a successful wallet deduction but a lost response, resulting in inconsistent data.

Even with circuit‑breaker frameworks like Hystrix, the fundamental issue remains: RPC calls should not be placed inside a transaction that expects strong consistency.

Asynchronous Message Push Inside Transactions

Using a message queue for inter‑service communication is common, but the asynchronous nature means the producer cannot know whether the consumer processed the message successfully. A naïve implementation pushes a message inside a transaction and then commits:

processOrder(){
  beginTransaction();
  queryOrder();
  pushMessageToWallet();
  updateOrderStatus();
  commitTransaction();
}

This approach suffers from two main problems:

If the message broker fails, the transaction rolls back, effectively making the broker a required service.

If the broker succeeds but a later SQL statement fails, the local transaction rolls back while the downstream service has already processed the message, causing data inconsistency.

Therefore, pushing messages asynchronously within a transaction is unreliable.

Industry Solutions

Common distributed‑transaction solutions include:

Two‑phase or three‑phase commit (2PC/3PC) – strong consistency but heavy performance cost.

Compensating transactions (TCC) – three operations (Try, Confirm, Cancel) reduce complexity but require extra code.

Message‑based transactions (e.g., RocketMQ) – pre‑message, local transaction, and confirm phases.

In practice, many teams adopt a simpler pattern based on a local message table.

Local Message Table Pattern

When using RabbitMQ (or similar), the solution stores messages in a local table that participates in the same database transaction as the business logic. After the transaction commits, a separate scheduler or real‑time trigger reads pending rows and pushes them to the broker.

Key steps:

Persist the message together with business updates inside a single transaction.

Commit the transaction.

Outside the transaction, query the local table and push messages, or let a scheduled job retry failed pushes.

Sample pseudo‑code for high‑throughput, loss‑tolerant pushes (no local table needed):

@Transactional(rollbackFor=RuntimeException.class)
public void process(){
  // business logic
  TransactionSynchronizationManager.getSynchronizations().add(
    new TransactionSynchronizationAdapter(){
      @Override
      public void afterCommit(){
        // push message here
      }
    }
  );
}

For scenarios that cannot tolerate loss, the flow includes writing to the local message table, committing, and then pushing in a separate step, with retry limits and alerting for stuck rows.

Idempotency Control

Idempotency ensures that repeated identical requests produce the same effect as a single request. Techniques used include:

Distributed locks (e.g., Redisson) around critical sections.

Business‑level duplicate checks (e.g., verify order number before creating).

Database unique constraints on logical keys.

Example of idempotent message consumption using a distributed lock:

listen(request){
  String lockKey = "msg:" + request.businessKey;
  RLock lock = redisson.getLock(lockKey);
  lock.lock();
  try{
    // business logic, de‑duplication, DB ops
  }finally{
    lock.unlock();
  }
}

Compensation Strategies

Compensation can be handled synchronously or asynchronously:

HTTP synchronous compensation – retry with exponential back‑off; short‑term retries are often ineffective if the downstream service is down.

Persist failed HTTP calls in a local retry table and schedule retries.

Asynchronous message consumption failures – let the consumer record the failed message in a local table and retry, with alerts after reaching a retry limit.

Asynchronous Message Disorder

Message ordering issues arise when multiple consumer threads process messages out of the original send order. Simple solutions:

Configure a single consumer thread to preserve FIFO ordering (supported by RabbitMQ).

Use HTTP calls for strictly ordered operations.

For cases where ordering is required for final presentation (e.g., repayment logs), attach a monotonically increasing sequence number or timestamp to each message and sort on the consumer side.

State‑Driven Asynchronous Messaging

A robust pattern combines pre‑processing, local message tables, and state‑driven workflows to achieve high consistency and performance. Example: a transfer operation where Service A deducts funds, writes a pre‑deduction record with a global SEQ_NO, pushes a message, Service B credits the target account and records the change, then Service B acknowledges success with another message. Service A finally updates the pre‑deduction record to “completed”. Periodic reconciliation and retry jobs ensure eventual consistency.

Conclusion

The most practical approach is to avoid strong‑consistency distributed transactions, favor BASE principles, and use message queues with a local message table to bind business updates and message pushes in the same transaction. Proper idempotency, compensation, ordering guarantees, and periodic reconciliation together provide a reliable yet scalable solution for microservice systems.