How to Achieve Distributed Transaction Consistency with Message Queues

In the distributed era, sharding databases is common; microservice systems usually use independent databases, making it difficult for the database itself to guarantee transactions, so business logic must handle them.

For example, in Alipay the balance and Huabei services likely operate on separate databases, and a transfer involves deducting from one service and adding to another, illustrating a distributed transaction.

Final consistency is typically achieved using a message middleware. The architecture is shown below:

User A initiates a transfer request to system A. System A first deducts money in its own database, then sends a message via the middleware to system B, which adds the money in its own database upon receipt.

The critical issue is that updating the database and sending the message are two separate operations, leading to two problematic scenarios:

Database update succeeds but message sending fails repeatedly.

Message sending succeeds but database update fails, leaving the message unrecoverable.

Because these operations are not atomic, combining them into a single transaction is not advisable.

To ensure atomicity, a practical approach is to introduce a message table. System A writes the message to this table instead of sending it directly to the middleware. A background process continuously reads messages from the table and forwards them to the middleware, retrying on failure.

This design guarantees that messages are not lost and their order is preserved, though duplicates may occur. System B must handle duplicates, for example by using a deduplication table to record successfully processed messages and achieve idempotency.

System B also needs to address two concerns:

Message loss: if B crashes before acknowledging a message, the middleware will redeliver it using an ACK mechanism.

Message duplication: ACK failures or repeated sending can cause the same message to be delivered multiple times.

By employing an ACK mechanism and a deduplication table, both systems can achieve reliable eventual consistency.

The overall architecture is illustrated below:

In summary, using eventual consistency with a message table, background processing, ACK handling, and deduplication provides a robust solution for distributed transaction problems, ensuring A does not lose messages and B processes them without loss and with idempotency.