Full-Stack Guide to Tracing RocketMQ Message Loss

Understanding Where Messages Disappear

RocketMQ messages can be lost in three stages: Producer → Broker , Broker storage , and Broker → Consumer . Each stage has typical loss scenarios, severity ratings and investigation difficulty.

Quickly Locate the Lost Segment

Step 1: Check Message Status – In the RocketMQ Dashboard, query the message ID or time range and examine the TrackType: CONSUMED: Verify consumer logs to confirm processing. CONSUMED_BUT_FILTERED: Subscription tag mismatch. NOT_CONSUME_YET: Consumer online but not pulling; check load‑balancing and queue allocation. NOT_ONLINE: Consumer down; message remains in the broker.

Not found: Message never entered the broker – production‑side loss.

Step 2: Compare Timestamps – Examine the three timestamps (send → store, store → consume). Normal delays are <10 ms for send‑to‑store and <50 ms for store‑to‑consume; larger values indicate broker slowness, network jitter, or consumer backlog.

Step 3: Use a Decision Table – Map the observed status to the most likely loss segment.

Production‑Side Loss Causes and Safeguards

One‑way send ( producer.sendOneway(msg)) – no acknowledgement; avoid for critical data.

Synchronous send with swallowed SendException – search logs for "send message failed".

Asynchronous send with empty onException implementation – inspect the SendCallback code.

Network interruption without retry – check broker logs for rejection codes.

DefaultMQProducer producer = new DefaultMQProducer("RELIABLE_PRODUCER_GROUP");
producer.setRetryTimesWhenSendFailed(3);               // sync send retry 3×
producer.setRetryTimesWhenSendAsyncFailed(3);           // async send retry 3×
producer.setRetryAnotherBrokerWhenNotStoreOK(true);    // switch broker on store failure
producer.setSendMsgTimeout(3000);                     // 3 s timeout
Message msg = new Message("ORDER_TOPIC", "TAG_A", "KEY_" + orderId,
    payload.getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult result = producer.send(msg);
if (result.getSendStatus() != SendStatus.SEND_OK) {
    log.error("Send failed, will retry or compensate: {}", result.getMsgId());
    saveFailMsgToDb(orderId, payload); // persist for later compensation
}

Broker‑Side Loss Causes and Safeguards

Async flush crash – check store.log for "flush disk error".

Async master‑slave not synchronized – monitor replication lag; delay > 1000 messages is risky.

Disk full or damaged – run df -h to inspect usage.

Message TTL too short – verify messageTTL setting on the topic.

# broker.conf
flushDiskType = SYNC_FLUSH   # write to disk before ack (≈40% slower, zero loss)
brokerRole = SYNC_MASTER   # wait for slave sync (recommended)
haSyncReplicas = 1          # number of sync replicas
persistent = true          # enable persistence
fileReservedTime = 72      # keep files for 72 h to avoid sudden full disk

Consumer‑Side Loss Causes and Defensive Config

Early ACK – message returned CONSUME_SUCCESS before business logic finishes.

Asynchronous processing – spawn a thread and immediately ack.

Business exception swallowed – catch block returns success.

Retry exhausted – after 16 attempts the message goes to the dead‑letter queue.

Subscription mismatch – producer tag differs from consumer tag.

consumer.registerMessageListener(new MessageListenerConcurrently() {
    @Override
    public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
            ConsumeConcurrentlyContext context) {
        for (MessageExt msg : msgs) {
            try {
                // 1. Business logic
                processOrder(msg);
                // 2. If success, framework auto‑commits offset
            } catch (Exception e) {
                log.error("Business failed, will retry: {}", msg.getMsgId(), e);
                return ConsumeConcurrentlyStatus.RECONSUME_LATER; // push to retry queue
            }
        }
        return ConsumeConcurrentlyStatus.CONSUME_SUCCESS; // manual ack
    }
});
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
((MQPushConsumer)consumer).setAutoCommit(false); // disable auto‑commit

Never start a new thread to process the message and immediately return CONSUME_SUCCESS – the message will be lost.

Recovery Strategies After Loss

P0 – Consume dead‑letter queue when consumer retries are exhausted. Subscribe %DLQ%ConsumerGroup, compensate and re‑store.

P1 – Producer compensation table for failed sends. A scheduled job scans mq_fail_msg and re‑sends.

P2 – Broker data recovery after broker failure. Restore commitlog from the slave's store directory.

P3 – Business reconciliation when other methods fail. Manually compare by time/unique ID and re‑push.

Monitoring and Alerting

# Prometheus alert example
- alert: RocketMQMessageAccumulation
  expr: rocketmq_message_accumulation_total > 10000
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.topic }} message backlog exceeds 10,000"

- alert: RocketMQDeadLetterQueue
  expr: rocketmq_dlq_message_count > 0
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "Dead‑letter queue contains messages, investigate immediately"

Final Takeaway

In most cases ( 99 % ) message loss is not a flaw in RocketMQ itself but stems from improper usage patterns such as unchecked send results, asynchronous flush without replication, or premature consumer acknowledgements.

Production: always check send status and configure retries.

Broker: use synchronous flush and master‑slave sync for critical workloads.

Consumer: employ manual ACK, idempotent processing, and avoid async thread‑hand‑off before ack.