Guaranteeing No Message Loss in RabbitMQ with Persistence and Confirm

Preface

Message middleware such as RabbitMQ, RocketMQ, and Kafka are widely used to handle high concurrency, peak shaving, and service decoupling. This article discusses how to guarantee that an order service successfully delivers messages to the MQ, using RabbitMQ as an example.

Problem Analysis

Simply sending a message and receiving a success response does not ensure the message is truly persisted. If the MQ server crashes before flushing messages from memory to disk, all in‑memory messages are lost, which is unacceptable for business continuity.

Persistence

Setting the durable flag to true when publishing messages makes them persistent, so they survive MQ restarts. However, messages may still be lost if the server crashes before the in‑memory buffer is flushed to disk.

Confirm Mechanism

RabbitMQ provides a confirm mechanism that notifies the producer whether a message was successfully persisted. An ack indicates success, while a nack indicates failure. This mechanism is asynchronous to maintain high throughput, but it cannot guarantee 100% durability because a crash may occur before the in‑memory message is flushed.

Pre‑persist Message + Scheduled Task

To overcome this, the order service first stores the message in Redis (or a database) with a status of "sending". After publishing, the confirm callback removes the Redis entry on ack. If nack occurs, the message can be retried or discarded based on business logic. A scheduled task periodically checks for messages still marked as "sending" and re‑publishes them, ensuring eventual delivery.

Idempotency Meaning

Idempotency means that repeated execution of the same operation under the same conditions yields the same result, which is crucial in distributed systems where duplicate messages may occur.

Why Idempotency is Needed

In a distributed architecture, services like order and inventory are deployed separately. Network failures can cause the order service to miss a successful inventory update, leading to duplicate deductions. Idempotent operations prevent such inconsistencies.

Optimistic Lock Solution

Borrowing the optimistic lock pattern from databases, each inventory update includes a version number. If the version has changed, the update is skipped, ensuring only one successful operation per request.

Unique ID + Fingerprint

Use the business entity's primary key as a unique ID and generate a fingerprint (e.g., timestamp + business code) for each operation. Insert a record with the unique ID and fingerprint; if the insert fails because the ID already exists, the operation is ignored, providing a simple deduplication mechanism.

Redis Atomic Operations

Leverage Redis atomic commands to mark operation completion. Challenges include ensuring atomicity between database writes and Redis updates, and deciding on synchronization strategies if data is only cached.