How to Build a Reliable Message Service for Distributed Transactions

Implement Reliable Message Service

First look at a simplified architecture diagram:

Core Process 1: Upstream Message Publishing

The business initiator sends a message to the message service, which stores it with status “Pending Confirmation”. If storage fails, the operation ends with a persistence failure. After receiving the storage result, the initiator executes its local transaction. If the local transaction succeeds, it calls the message service to confirm the message, updating the status to “Pending Send”. If the local transaction fails, it calls the service to delete (logically) the message, setting the status to “Rolled Back”.

When the status is the first type, the message service sends the message to the MQ and updates the status to “Sent”.

Note: Updating the message status and delivering the message to the MQ must occur in the same method within a local transaction to guarantee that both succeed or both fail.

If the status update fails, an exception is thrown to roll back the transaction; if MQ delivery fails, the exception is caught and re‑thrown to trigger a rollback.

Core Process 2: Passive Service Consumes Message

The passive service subscribes to the topic and waits for MQ delivery. Upon receiving a message, it executes its local business logic. If the local operation succeeds, the passive service calls the message service to indicate success, and the service sets the message status to “Completed”. Throughout this process, the passive service strives to push the business toward a final state, ensuring either successful completion or definitive failure and notifying the message service accordingly.

1. How to Ensure No Message Loss – Reliable Delivery

Various scenarios are considered. If publishing the “Pending Confirmation” message fails, the initiator immediately perceives the failure and aborts. If, after the local transaction, notifying the reliable message service fails (or the service fails to deliver to the MQ), the message remains in the “Pending Confirmation” state.

A scheduled task scans messages in the “Pending Confirmation” state. For each such message, the service invokes a callback interface exposed by the initiator to query the local transaction status. If the initiator reports success, the service updates the message status to “Pending Send”, delivers it to the MQ, and then sets the status to “Sent”. If the initiator reports failure, the service logically deletes the message, setting the status to “Rolled Back”.

2. How to Ensure Passive Side Receives Message 100% – Redelivery

If a message has been sent but the passive consumer fails to process it (e.g., consumption failure), a scheduled task rescans messages that remain in the “Sent” state without transitioning to “Completed”. After a configurable interval (e.g., one minute), the service re‑delivers the message to the MQ.

Redelivery requires the business logic to be idempotent so that multiple deliveries result in only one complete execution.

From message sending to consumption, the message state stays consistent with the local transaction, achieving eventual consistency across the distributed workflow.

Summary

By leveraging existing message‑queue middleware, we can construct a reliable message service that guarantees the upstream transaction and downstream message delivery either both succeed or both fail, providing an eventual‑consistency solution for distributed systems.