How to Prevent Message Loss in RocketMQ: Producer, Broker, and Consumer Strategies

Message loss points

RocketMQ can lose messages at three stages: the producer (network failure prevents the broker from receiving the message), the broker storage (messages kept only in memory are lost if the broker crashes before flushing to disk), and the consumer (automatic acknowledgment, consumption timeout, or processing failures after a successful ACK cause loss).

Producer‑side mitigation

RocketMQ offers three sending modes.

Synchronous send blocks the producer until the broker returns an ACK. Example code checks the SendResult; if sendResult.getSendStatus() != SendStatus.SEND_OK, the business operation can be logged, retried, or rolled back.

public void handleBusiness() {
  // core business logic
  SendResult sendResult = producer.send(msg);
  if (sendResult.getSendStatus() != SendStatus.SEND_OK) {
    // log, manual intervention or retry
  }
}

Asynchronous send returns immediately; a SendCallback reports success or failure. The callback can log success or perform compensation (e.g., write to a local log or database) on exception. Retries and manual compensation are required to guarantee eventual delivery.

producer.send(msg, new SendCallback() {
  @Override
  public void onSuccess(SendResult sendResult) {
    // record success
  }
  @Override
  public void onException(Throwable e) {
    // compensation logic (e.g., local persistence)
  }
});

Oneway is fire‑and‑forget: the producer sends the message without waiting for any broker response. Suitable for scenarios that can tolerate occasional loss, such as log collection, monitoring metrics, or non‑critical notifications.

// Oneway send
producer.sendOneway(msg);

Broker‑side mitigation

By default, the broker writes incoming messages to memory and flushes them to disk asynchronously. If the broker crashes before the flush, the message is lost. Enabling synchronous disk flush forces the broker to acknowledge only after the message is persisted.

# broker.conf
flushDiskType=SYNC_FLUSH

In a master‑slave deployment, the master writes to its own disk and replicates to the slave asynchronously. If the master fails before replication completes, the slave does not have the message, leading to loss. Setting the broker role to SYNC_MASTER makes the master wait for the slave’s acknowledgment, ensuring both nodes have the message before returning ACK.

# broker.conf
brokerRole=SYNC_MASTER

Consumer‑side mitigation

Typical loss scenarios include business exceptions after the consumer has already returned CONSUME_SUCCESS, or using asynchronous thread pools where the main thread reports success before the async task completes. The core principle is to acknowledge only after business processing succeeds, allowing RocketMQ to retry on failure.

Switching from automatic to manual acknowledgment is done by registering a MessageListenerConcurrently that returns ConsumeConcurrentlyStatus.CONSUME_SUCCESS on success and ConsumeConcurrentlyStatus.RECONSUME_LATER on exception.

// Disable auto‑commit, use manual ack
consumer.registerMessageListener(new MessageListenerConcurrently() {
  @Override
  public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
    try {
      // business processing
      process(msgs);
      return ConsumeConcurrentlyStatus.CONSUME_SUCCESS; // manual ack
    } catch (Exception e) {
      return ConsumeConcurrentlyStatus.RECONSUME_LATER; // trigger retry
    }
  }
});

Failed consumption is retried up to 16 times by default. After exceeding the retry limit, the message is moved to a dead‑letter queue named %DLQ%+ConsumerGroup. Monitoring this queue and performing manual compensation for its contents is required to avoid permanent loss.

Key measures to prevent loss

Producer side – use synchronous send for critical messages; for asynchronous send, combine with retry and compensation logic; avoid Oneway for messages that must not be lost.

Broker side – enable synchronous disk flush ( SYNC_FLUSH) and, in master‑slave clusters, use synchronous replication ( SYNC_MASTER).

Consumer side – switch to manual acknowledgment, implement idempotent processing, and monitor dead‑letter queues for messages that exceed retry limits.