RocketMQ Production Pitfalls: Solving Message Backlog, Loss, Duplicates & Dead Letters

When to Choose RocketMQ — and When Not To

The team adopted RocketMQ after completing a sharding rewrite for an e-commerce platform handling hundreds of millions of orders. In a microservice architecture, databases handle storage while message queues manage asynchronous link flow.

Three Core Use Cases in the Mall

Business async decoupling — Order creation only persists the order; coupon issuance, SMS, logistics initialization, and user behavior stats are offloaded to MQ, shortening the main-path RT and boosting throughput.

Traffic peak shaving — During flash sales, MQ buffers the surge; consumers drain at a pace the downstream DB and services can sustain, preventing instant overload.

Distributed transaction eventual consistency — Combined with local message tables or RocketMQ transaction messages, it coordinates cross-service consistency (e.g., order creation + inventory deduction).

Scenarios Where MQ Is Forbidden

Strong real-time synchronous response required (e.g., payment result verification).

Core links that cannot tolerate even brief inconsistency.

Tiny message volume where MQ adds unnecessary operational complexity.

Architectural iron law: MQ solves problems; don't use MQ just for the sake of using MQ.

<!-- rocketMQ client core dependency -->
<dependency>
  <groupId>org.apache.rocketmq</groupId>
  <artifactId>rocketmq-spring-boot-starter</artifactId>
  <version>2.3.0</version>
</dependency>

Message Sending Phase: Eliminating Message Loss

Most message loss isn't caused by the broker — it's the sender mishandling the send result. Many developers assume send() guarantees 100% delivery.

Real Production Incident

Early version used async send for coupon issuance with a callback that only logged. A network glitch caused mass delivery failures; no failure records were persisted. Business assumed success, thousands of users missed coupons, requiring manual script compensation.

Three Send Modes Compared

Sync send — Blocks until broker returns ack, gets explicit status code. Used for order creation, inventory changes — core business, production default. Risk: occupies main thread; evaluate interface latency under high concurrency.

Async send — Callback receives result, non-blocking. Used for SMS, in-app notifications, behavior stats — non-core. Risk: Callback MUST handle failure logic, not just log .

OneWay — Returns immediately, no broker ack wait. Used for log reporting, tracking stats. Risk: FORBIDDEN for core business; network jitter loses messages easily .

Production rule: Order, inventory, coupon messages — absolutely no OneWay.

Complete Sync Send Example (Order Creation)

// Mall order creation sync send full example
try {
  Message<String> message = MessageBuilder.withPayload(JSON.toJSONString(orderCreateDTO))
    .setHeader(RocketMQHeaders.TOPIC, "order_create_topic")
    .build();
  SendResult sendResult = rocketMQTemplate.syncSend("order_create_topic", message, 3000);
  // Strictly verify delivery status
  if (!SendStatus.SEND_OK.equals(sendResult.getSendStatus())) {
    // Delivery failed, write to local message table, scheduled retry compensation
    saveLocalMsgRecord(orderId, JSON.toJSONString(orderCreateDTO));
    log.warn("Order message delivery not OK orderId:{},status:{}", orderId, sendResult.getSendStatus());
  }
} catch (MQClientException | RemotingException e) {
  // Broker unreachable, network exception, persist local message for compensation
  saveLocalMsgRecord(orderId, JSON.toJSONString(orderCreateDTO));
  log.error("Order message send exception orderId:{}", orderId, e);
}

Send Failure Fallback: Local Message Table

On send exception, don't drop the message. Persist to a local message table; a scheduled job scans un-sent messages and retries. After exceeding a retry threshold, alert for manual intervention — eliminating loss at the source.

Broker-side: Enable flush policy; core business uses SYNC_FLUSH synchronous flush to avoid disk loss on broker power loss.

Consumer Core Challenge: Duplicate Consumption — Business Must Be Idempotent

RocketMQ guarantees at-least-once delivery, not exactly-once. If business logic succeeds but offset commit fails due to network jitter, the broker re-delivers, causing duplicates.

Real Incident

During a promotion, order messages were re-delivered. Consumer lacked idempotency; the same order executed coupon issuance twice, some users received two coupons — direct financial loss.

Three Production Idempotency Solutions (Pick by Business)

DB unique index (preferred) — Consumer passes business unique ID (orderId), unique index on table. Duplicate insert throws duplicate key error, blocking repeat. Best for insert-heavy flows.

Dedicated idempotency record table — Separate table stores business ID + processing state. Before consumption, query if orderId already processed; if not, execute logic then write record. Suits complex multi-table updates.

Business state machine check — Rely on inherent state flow; if order status is FINISH, skip consumption. Fits businesses with complete state transitions.

Complete Consumer Pseudocode

@RocketMQMessageListener(topic = "order_create_topic", consumerGroup = "mall-order-consumer-group")
public class OrderCreateConsumer implements RocketMQListener<OrderCreateDTO> {
  @Override
  public void onMessage(OrderCreateDTO dto) {
    String orderId = dto.getOrderId();
    // 1. State machine check — already finished, return
    Order order = orderMapper.selectById(orderId);
    if (OrderStatus.FINISH.getCode().equals(order.getStatus())) {
      log.info("Duplicate consumption, skip orderId:{}", orderId);
      return;
    }
    // 2. Idempotency table second check — prevent concurrent re-entry
    IdempotentRecord record = idempotentMapper.selectById(orderId);
    if (record != null) {
      log.info("Idempotency hit, skip orderId:{}", orderId);
      return;
    }
    // Execute business: issue coupon, init logistics
    couponService.sendCoupon(orderId);
    logisticsService.initLogistics(orderId);
    // Write idempotency record
    idempotentMapper.insert(new IdempotentRecord(orderId, LocalDateTime.now()));
  }
}

Critical: Idempotency logic MUST sit at the very start of consumption — never after business execution.

Message Backlog: Top Production Incident During Promotions

Real Postmortem (618 Sale)

A consumer service deployed a bug: third-party logistics HTTP call occasionally timed out and threw. Messages retried endlessly; consumption lagged far behind production. Within minutes 800,000+ messages piled up. Coupon issuance and logistics init delayed; users waited 30+ minutes for coupons; massive complaints.

Common Root Causes

Heavy consumption logic — slow SQL, synchronous external HTTP calls, high per-message latency.

Consumer code throws exceptions, triggering infinite retries, stalling progress.

Too few consumer instances — capacity can't match upstream throughput.

Downstream DB or third-party API failure blocks consumer threads.

Consumer group misconfiguration — consumer count less than topic read/write queues, underutilizing parallelism.

Emergency Response Steps

Fix the consumer bug first — resolve the root exception cause.

Horizontal scale consumer instances — boost total consumption capacity to drain backlog fast.

If backlog hits millions , spin up a temporary consumer group dedicated to historical stock; original group handles new inflow, preventing new messages from being dragged down.

Optimize consumption logic — convert internal sync third-party calls to async, reduce per-message cost.

Best practice: Keep onMessage lightweight; avoid heavy I/O or synchronous remote calls inside.

Early Warning Metrics

Monitor and alert on: message pile-up count, per-message consumption latency, consumption failure count, retry message count . Don't wait for user complaints to discover backlog.

Retry Mechanism & Dead Letter Queue (DLQ) — Most Teams Overlook This

On consumption exception, RocketMQ doesn't drop the message; it retries at increasing intervals. Once retries exhaust, the message lands in the dead letter queue (DLQ) . DLQ messages are never auto-consumed again — if unattended, the business message is permanently lost.

Mall scenario: Order message in DLQ means user coupon and logistics init never happen — direct business impact.

Mall's Complete DLQ Handling

Dedicated listener on DLQ topic — on receipt, immediately fire alert (DingTalk/WeCom) to devs.

Persist full DLQ message body + exception details to DB for forensics.

Admin UI allows manual re-delivery after root cause fix.

Major hidden risk: Many production teams ignore DLQ entirely after MQ launch.

Supplement: Never use infinite retry in production. Set a sensible max retry count; otherwise faulty messages loop forever, hogging consumer threads and worsening backlog.

Other Hidden Production Pitfalls

6.1 Don't Mix Consumer Groups

Different businesses = different consumerGroup. Same group = load balancing across instances. Sharing a group across businesses causes misrouted messages, loss, and duplicates.

6.2 Keep Message Body Small

Single message < 16 KB . Large messages degrade broker performance. Don't stuff big objects; pass business IDs, let consumer fetch full data from DB.

6.3 Use Ordered Messages Sparingly

Ordered messages perform poorly — queue count forced to 1, killing concurrency. Only use for true strong-ordering needs. Mall uses them only for a few order status change flows.

6.4 Don't Abuse Transaction Messages

RocketMQ transaction messages have edge-case traps. Prefer local message table for eventual consistency — simpler, more controllable, easier to debug.

6.5 Tune Consumer Thread Pool

Adjust thread count per consumption latency. More threads ≠ better; excess threads hammer DB and downstream APIs.

Architect's Battle Summary

1. Core business forbids OneWay; sender catches exceptions, pairs with local message table for compensation — stop loss at source; broker core topics enable sync flush. 2. MQ guarantees at-least-once; consumer MUST implement idempotency — never assume single consumption. 3. Keep consumption logic light; avoid heavy sync remote calls inside consumer to prevent lag. 4. Full monitoring & alerting: pile-up, failures, DLQ — all alerted; pre-promotion load test to gauge consumption watermark. 5. DLQ must have listener + alert; DLQ ≠ discarded — requires human intervention. 6. Separate consumer groups per business, cap message size, use ordered/transaction messages cautiously, favor simple controllable solutions.

Message queues are easy to start but hide deep production traps. Many faults only surface under real high-concurrency load, not in test envs. Rigorous, standardized rollout is the only way to stabilize the entire async business chain.