Understanding RocketMQ’s Three Reliability Checkpoints

RocketMQ’s reliability design covers every stage of message flow but does not guarantee zero loss; effectiveness depends on proper configuration and code.

Three checkpoints

A message passes three checkpoints: (1) producer to broker (send), (2) broker persistence to disk (store), and (3) consumer pull and offset commit (consume). Failure at any checkpoint can cause loss.

Send checkpoint: producer to broker

Sending modes

Synchronous send : blocks until the broker responds, allowing the producer to detect failure via the return value and retry. Default internal retry count is 2 (total attempts 3). Suitable for scenarios that cannot tolerate loss such as transactions or payments.

Asynchronous send : does not block; a callback receives the send result. Internal network‑level retry exists but does not switch brokers. Suitable for latency‑sensitive scenarios that still require reliability.

One‑way send : fire‑and‑forget, no failure detection or internal retry. Suitable for log collection or metric reporting where loss is acceptable.

Pitfall : asynchronous send retries stay on the same broker, while synchronous send retries switch to a different broker, reducing fault‑tolerance for the asynchronous mode.

Send retry and fault avoidance

The synchronous send retry logic resides in DefaultMQProducerImpl.sendDefaultImpl. Total attempts = 1 + retryTimesWhenSendFailed (default 2, maximum 3). Each retry passes the previous broker name to avoid that broker.

On failure, the client updates an internal fault table managed by MQFaultStrategy. Brokers with response time >2000 ms are marked unavailable for 120 s; network exceptions mark a broker unavailable for 10 min. During the penalty period, other brokers are preferred. This mechanism is disabled by default; enable it with: producer.setSendLatencyFaultEnable(true); Enabling the setting improves success rates in multi‑broker clusters by avoiding problematic brokers.

Store checkpoint: broker persistence

After a broker receives a message, it writes to the CommitLog in the OS PageCache. If the machine loses power before the data is flushed to disk, the message is lost. RocketMQ provides two flush strategies.

Sync flush

Implemented by GroupCommitService. The producer thread blocks until the flush thread writes the data to disk. A read/write queue exchange reduces lock contention. Each request may trigger up to two flushes to handle file‑boundary cases. Sync flush guarantees that every message is on disk before success is returned, but adds significant I/O pressure and lowers throughput.

Async flush

Implemented by FlushRealTimeService. The producer returns immediately; a background thread flushes every 500 ms or when at least four pages (16 KB) accumulate. A full flush occurs at least every 10 s. Async flush offers higher throughput but can lose messages if the broker crashes between flushes.

Master‑slave synchronization

HAService

handles replication. The master records the highest offset replicated to the slave; a slave is considered healthy if the connection is alive and the lag ≤256 MB (default).

Sync replication (SYNC_MASTER) : the master waits for at least one slave acknowledgment before returning success.

Async replication (ASYNC_MASTER) : the master returns immediately; the slave receives data asynchronously.

Reliability‑performance combinations

Sync flush + sync replication: no message loss, lowest throughput, suitable for financial‑grade transactions.

Sync flush + async replication: no loss unless the master disk fails, lower throughput, suitable for core business where occasional loss is acceptable.

Async flush + sync replication: possible loss if the master crashes, higher throughput, suitable for important business with throughput requirements.

Async flush + async replication: loss of PageCache data on master crash, highest throughput, suitable for log‑type workloads tolerant of minor loss.

Most online systems use “async flush + async replication” or “async flush + sync replication”. Sync flush can become a bottleneck under high concurrency.

Corresponding broker configuration example:

flushDiskType=ASYNC_FLUSH
brokerRole=SYNC_MASTER

Consume checkpoint: broker to consumer

Even with reliable storage, the consumer can lose messages, e.g., when the offset is committed before processing finishes.

Reliability principle: consume first, commit offset only after successful processing.

Consume acknowledgment mechanisms

In clustering mode, a failed message is sent back to the broker via sendMessageBack, entering a retry queue. If sendMessageBack itself fails (e.g., network outage), the message is placed in a local failure list and retried after 5 seconds.

In broadcasting mode, failed messages are discarded with a warning log. Users needing no loss must implement their own retry logic.

Consume retry delay strategy

When a message is sent back, the broker determines the next delivery delay based on the current retry count. Delay level = 3 + retryCount. RocketMQ defines 18 levels ranging from 1 second to 2 hours. The first retry starts at level 3 (10 seconds) and increases. The schedule is:

Retry 1 – 10 s

Retry 2 – 30 s

Retry 3 – 1 min

Retry 4 – 2 min

Retry 5 – 3 min

Retry 6 – 4 min

Retry 7 – 5 min

Retry 8 – 6 min

Retry 9 – 7 min

Retry 10 – 8 min

Retry 11 – 9 min

Retry 12 – 10 min

Retry 13 – 20 min

Retry 14 – 30 min

Retry 15 – 1 h

Retry 16 – 2 h

Dead‑letter queue

If the retry count exceeds the maximum (default 16, configurable via SubscriptionGroupConfig.retryMaxTimes), the message is moved to a dead‑letter queue whose topic name is %DLQ% plus the consumer group name. Messages in the DLQ are not consumed automatically and require manual handling. Production systems should monitor the DLQ, e.g., via RocketMQ Dashboard or a dedicated consumer that raises alerts or performs compensation.

Offset commit timing

After processing, the client removes consumed messages from the local ProcessQueue, computes the smallest remaining offset, and updates the broker. If an early message keeps failing, later successful messages cannot advance the offset, causing them to be re‑delivered after a restart. This reflects RocketMQ’s at‑least‑once semantics; business code must be idempotent.

Production‑grade deployment checklist

Producer side

Use synchronous send and verify that SendStatus is SEND_OK.

Set retryTimesWhenSendFailed to 3 (one extra retry).

Enable sendLatencyFaultEnable for more effective retries.

Configure sendMsgTimeout (e.g., 5 seconds).

Broker side

Core business: async flush + sync replication for a balance of performance and reliability.

Financial scenarios: sync flush + sync replication for zero loss.

Consumer side

When business logic throws an exception, return RECONSUME_LATER instead of swallowing the exception and returning CONSUME_SUCCESS.

Implement idempotency using a unique message ID or business key.

Adjust the maximum retry count according to processing time requirements.

Monitor the dead‑letter queue and set up alerts.

Conclusion

Message reliability is a full‑chain problem; RocketMQ provides safeguards at each stage, each with its own performance cost. Sync flush guarantees disk durability but reduces throughput; sync replication adds latency. Strengthening the weakest checkpoint yields practical reliability without sacrificing performance.