Ensuring Consistent Money Transfers: Distributed Transactions & Message Queues Explained

Background and Problem Statement

When a user transfers money from an Alipay account to a YuEBao account, the debit and credit operations occur on two separate tables. If the system crashes after debiting the source account but before crediting the destination, the data becomes inconsistent. This issue appears in many domains, such as e‑commerce order placement and advertising fee deduction.

Local Transaction Solution

In a single‑machine setup where both tables reside in the same database instance, a local transaction can guarantee atomicity:

BEGIN TRANSACTION;
UPDATE A SET amount = amount - 10000 WHERE userId = 1;
UPDATE B SET amount = amount + 10000 WHERE userId = 1;
COMMIT;

Spring developers can achieve the same effect with a single annotation:

@Transactional(rollbackFor = Exception.class)
public void update() {
    updateATable();
    updateBTable();
}

Why Local Transactions Fail at Scale

When the two tables are stored on different physical nodes, a local transaction cannot span both databases. A distributed transaction mechanism is required.

Two‑Phase Commit (2PC)

2PC introduces a coordinator (TC) and multiple participants (Si). The protocol proceeds as follows:

Client sends a begin request to the coordinator.

TC writes a prepare log entry and sends prepare messages to all participants.

Each participant executes its local transaction, writes a log entry, and replies YES or NO.

If all replies are YES, TC sends commit to every participant; otherwise it sends abort .

Logging ensures recovery after crashes. However, 2PC suffers from high latency (multiple network round‑trips) and long lock holding times, making it unsuitable for high‑concurrency services.

Message‑Queue Based Eventual Consistency

Instead of a strict distributed transaction, the system can treat the message that records the credit operation as a voucher. The voucher is persisted reliably before the credit is performed, achieving eventual consistency.

1) Tightly Coupled Approach

The message is stored in the same database transaction that updates the debit account:

BEGIN TRANSACTION;
UPDATE A SET amount = amount - 10000 WHERE userId = 1;
INSERT INTO message(userId, amount, status) VALUES (1, 10000, 1);
COMMIT;

After the transaction commits, a real‑time messaging service forwards the message to the credit side, which then acknowledges and deletes the record.

2) Decoupled Approach

The message is written first, and the debit transaction is committed only after the messaging service confirms that the message has been stored:

Before committing the debit, the producer asks the messaging service to store the message (no actual send yet).

After the debit transaction succeeds, the producer confirms the send, causing the message to be delivered.

If the debit rolls back, the producer cancels the pending message.

To handle cases where the confirmation step fails, a periodic reconciliation job checks the message status and updates it accordingly.

Handling Duplicate Message Delivery

Duplicate delivery can cause the credit side to add money twice. The solution is to maintain a message_apply table that records processed message IDs. Before applying a message, the consumer checks this table; if the ID exists, the message is discarded.

FOR EACH msg IN queue
BEGIN TRANSACTION;
SELECT COUNT(*) INTO cnt FROM message_apply WHERE msg_id = msg.msg_id;
IF cnt = 0 THEN
    UPDATE B SET amount = amount + 10000 WHERE userId = 1;
    INSERT INTO message_apply(msg_id) VALUES (msg.msg_id);
END IF;
COMMIT;
END FOR

Large Transaction = Small Transactions + Asynchrony

A big transaction can be split into two smaller ones: a local debit transaction and an asynchronous message that triggers the credit. The system must ensure that both succeed or both fail. Two ordering options exist:

Send message first: If the message succeeds but the debit fails, the credit will be applied without a corresponding debit.

Debit first: If the debit succeeds but the message fails, money is deducted without being credited.

Both problems can be mitigated by embedding the message send inside the same local transaction (tightly coupled) or by using idempotent processing on the consumer side.

RocketMQ Transactional Messaging

RocketMQ implements transactional messaging with three phases: send a prepared message, execute the local transaction, then send a commit/abort based on the transaction outcome.

// Prepare phase
TransactionMQProducer producer = new TransactionMQProducer("groupName");
producer.setTransactionCheckListener(new TransactionCheckListenerImpl());
producer.start();
Message msg = new Message("TopicTransaction", "TagA", "KEY1", "Hello RocketMQ".getBytes());
SendResult result = producer.sendMessageInTransaction(msg, tranExecuter, null);
producer.shutdown();

The TransactionListener decides whether to commit, rollback, or leave the state unknown. RocketMQ periodically checks unknown states and invokes the listener to resolve them.

Practical Demo

The article provides a complete demo:

TransactionProducer creates a TransactionMQProducer, registers a TransactionListenerImpl, and sends ten messages.

TransactionListenerImpl implements executeLocalTransaction (simulated business logic) and checkLocalTransaction (used by RocketMQ for unknown states).

Consumer uses DefaultMQPushConsumer with an ordered listener to process committed messages.

Running the demo shows that only one of the four produced messages reaches the consumer because only one transaction was committed.

Ensuring End‑to‑End Consistency

Even with transactional messaging, failures can still leave the system in an inconsistent state. The article proposes a reconciliation topic that records every transaction and its outcome. A downstream accounting service consumes this topic, compares the producer’s debit log with the consumer’s credit log, and triggers compensating actions (e.g., rollback) when mismatches are detected.

Overall, the article demonstrates how to move from naive local transactions to robust distributed transaction patterns using 2PC, message‑queue based eventual consistency, duplicate‑message protection, and reconciliation mechanisms, with concrete RocketMQ code samples.