How to Prevent Message Loss and Ensure Reliable Delivery in Distributed Systems

1. Detecting Message Loss

Use sequential IDs injected by the producer and verified by the consumer; most MQ clients support interceptors to add IDs before sending and check continuity on receipt. In distributed setups, assign IDs per partition and include a producer identifier when multiple producers are used.

2. Ensuring Reliable Delivery

Production Stage

The producer receives an acknowledgment from the broker; capture exceptions and retry if needed.

try {
    producer.send(record).get();
    System.out.println("Message sent successfully");
} catch (Exception e) {
    System.out.println("Message sending failed");
    e.printStackTrace();
}

Asynchronous Send

producer.send(record, new Callback() {
    @Override
    public void onCompletion(RecordMetadata metadata, Exception exception) {
        if (metadata != null) {
            System.out.println("Message sent successfully");
        } else {
            System.out.println("Message sending failed");
            exception.printStackTrace();
        }
    }
});

Storage Stage

Configure the broker to flush to disk synchronously (e.g., RocketMQ flushDiskType=SYNC_FLUSH) or replicate messages to multiple nodes so that a single broker failure does not cause loss.

Consumption Stage

Process the business logic first, then acknowledge the message manually to avoid premature ACKs.

@RabbitListener(bindings = @QueueBinding(...))
public void onMessage(@Payload Order order, @Headers Map<String,Object> headers, Channel channel) throws Exception {
    // business logic
    System.out.println("Consumer: " + order);
    Long deliveryTag = (Long) headers.get(AmqpHeaders.DELIVERY_TAG);
    channel.basicAck(deliveryTag, false);
}

3. Handling Duplicate Messages

Most MQs provide at‑least‑once delivery, so duplicates are inevitable. Implement idempotent consumption using unique database constraints, conditional updates, version checks, or a token table that records processed message IDs.

4. Dealing with Message Backlog

Optimize producer performance by adjusting batch size and concurrency; optimize consumer performance by scaling instances and matching the number of partitions. When a backlog spikes, determine whether the production rate increased or the consumption rate decreased, then scale or temporarily degrade non‑critical traffic.

5. Distributed Transactions with Transactional Messages

Use half‑messages: the producer sends a pending message, executes the local DB transaction, then commits or rolls back the message. RocketMQ adds a transaction‑check callback so the broker can query the producer for the final state if the commit response is lost.

Consumers only see the message after the transaction is committed, guaranteeing atomicity between the database write and the message publish.