Skip to content
MQ Message Backlog Crisis: 3-Phase Production Survival Guide (Mitigate → Diagnose → Prevent)

MQ Message Backlog Crisis: 3-Phase Production Survival Guide (Mitigate → Diagnose → Prevent)

This article presents a three-phase framework for handling MQ message backlogs in production: immediate mitigation via consumer scaling, business degradation, and message dumping; systematic root cause analysis across producer, broker, and consumer layers; and long-term prevention through traffic control, async processing, monitoring, and automated dead-letter handling, with RocketMQ code examples.

Rain's Path to Java Mastery
Rain's Path to Java Mastery
Rain's Path to Java Mastery

Core Principle: Mitigate First, Investigate Second, Prevent Last

The fundamental rule when facing an MQ message backlog is "mitigate first, investigate second, prevent last" . The top priority is to guarantee core business availability immediately, then gradually locate and resolve the root cause. Never spend half an hour diagnosing while the business remains down.

Phase 1: Emergency Mitigation (Complete Within 10 Minutes)

This is the highest priority for any production incident — restore business first.

Emergency Flowchart

Key Mitigation Actions

1. Scale Consumers First

Core constraint: In Kafka/RocketMQ, a single partition can only be consumed by one consumer thread. Therefore, the maximum number of consumer threads equals the number of partitions.

Action: Temporarily increase the number of consumer group instances, then raise the core thread pool size of each instance.

2. Degrade Non-Core Business

Immediately stop consuming non-core messages such as logs, statistics, and pushes. Release CPU, memory, and database connections for core business.

Example: During an e-commerce flash sale, halt consumption of user behavior analysis and product recommendation messages.

3. Temporary Message Dump

When backlog reaches million-level or above , direct consumption can drag down the entire MQ cluster.

Solution: Write a simple script to dump messages to Redis, MySQL, or OSS first, then replay them slowly after the traffic peak.

4. Skip Dead-Letter Messages

If a large number of repeatedly failing dead-letter messages exist, skip them temporarily to avoid blocking normal message consumption.

Handle them later by pulling from the dead-letter queue manually or automatically.

Phase 2: Root Cause Analysis (Start Immediately After Business Recovery)

Investigate from three directions — producer, MQ cluster, consumer — 90%+ of issues originate on the consumer side.

Common Root Cause Lookup

Producer traffic spike: Produce TPS suddenly jumps 5-10x, consumption can't keep up. Check monitoring panel for produce rate curve; look for flash sales, crawlers, or batch jobs.

Consumer performance bottleneck: Consume TPS far below produce TPS, high CPU/IO. Check consumer logs for slow SQL / third-party timeouts; use jstack to see if threads are blocked on DB calls.

Consumer logic anomaly: Consume speed drops to zero, massive error logs. Check for infinite loops, null pointers, idempotency issues; verify if duplicate consumption causes dead loops.

MQ cluster failure: All consumer groups backlogged, produce also slows. Check broker disk usage, CPU, memory; look for broker crashes or network partitions.

Oversized message body: Single message exceeds 1MB, network transfer slow. Inspect message size distribution; check for large object serialization issues.

Often Overlooked Pitfalls

Consumer database connection pool exhaustion , causing all consumer threads to wait for connections.

Third-party interface timeout set too long (e.g., 30 seconds), blocking consumer threads for extended periods.

Consumer thread pool queue too long , causing tasks to pile up in memory, making consume speed appear slow.

Core Code Implementations

1. High-Performance Consumer Thread Pool Config (RocketMQ)

Key highlights: dynamic thread sizing based on partition count, JDK 21 virtual threads for massive concurrency, SynchronousQueue to avoid OOM and task pile-up, batch consumption (32 messages per pull) to reduce network I/O, and a retry strategy (max 3 retries) to prevent dead loops.

@Configuration
public class RocketMQConsumerConfig {
    // Highlight 1: Dynamic thread config based on partition count to avoid waste
    // Assuming topic has 16 partitions, each instance max 16 consume threads
    @Value("${rocketmq.consumer.thread-nums:16}")
    private int consumerThreadNums;

    @Bean(initMethod = "start", destroyMethod = "shutdown")
    public DefaultMQPushConsumer orderConsumer() throws MQClientException {
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("order_consumer_group");
        consumer.setNamesrvAddr("127.0.0.1:9876");
        consumer.subscribe("order_topic", "*");

        // Highlight 2: Use JDK 21 virtual threads to boost concurrency
        ThreadFactory virtualThreadFactory = Thread.ofVirtual()
                .name("order-consumer-", 0)
                .factory();

        // Highlight 3: Proper thread pool params to avoid OOM and pile-up
        consumer.setConsumeThreadMin(consumerThreadNums);
        consumer.setConsumeThreadMax(consumerThreadNums);
        // Critical: queue size 0, use SynchronousQueue for direct handoff
        consumer.setConsumeMessageThreadPoolQueue(new SynchronousQueue<>());
        // Use virtual thread factory
        consumer.setConsumeThreadFactory(virtualThreadFactory);

        // Highlight 4: Batch consumption, pull 32 messages at once to reduce network IO
        consumer.setConsumeMessageBatchMaxSize(32);
        consumer.setPullBatchSize(32);

        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
                    ConsumeConcurrentlyContext context) {
                try {
                    // Batch process messages
                    orderService.batchProcessOrders(msgs);
                    return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
                } catch (Exception e) {
                    log.error("消费消息失败", e);
                    // Highlight 5: Retry strategy, max 3 retries to avoid dead loop
                    if (context.getReconsumeTimes() >= 3) {
                        log.error("消息重试超过3次,转入死信队列: {}", msgs);
                        return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
                    }
                    return ConsumeConcurrentlyStatus.RECONSUME_LATER;
                }
            }
        });
        return consumer;
    }
}

2. Message Temporary Dump Tool (Emergency Mitigation Core Code)

Key highlight: write to dump storage first, then commit offset, guaranteeing zero message loss.

@Component
public class MessageDumpService {
    @Autowired
    private StringRedisTemplate redisTemplate;

    // Highlight: write dump first, then commit offset, ensuring no message loss
    public void dumpMessages(List<MessageExt> msgs) {
        String dumpKey = "mq:dump:order:" + System.currentTimeMillis();
        List<String> messageBodies = msgs.stream()
                .map(MessageExt::getBody)
                .map(String::new)
                .toList();

        // 1. Atomic write to Redis
        redisTemplate.opsForList().rightPushAll(dumpKey, messageBodies);
        // 2. Set expiry to avoid memory leak
        redisTemplate.expire(dumpKey, Duration.ofDays(7));

        log.info("成功转储{}条消息到Redis,key: {}", msgs.size(), dumpKey);
    }

    // Replay messages later
    public void replayMessages(String dumpKey) {
        while (true) {
            // Pop 10 messages each time
            List<String> messages = redisTemplate.opsForList().leftPop(dumpKey, 10);
            if (CollectionUtils.isEmpty(messages)) {
                break;
            }
            try {
                orderService.batchProcessOrderBodies(messages);
            } catch (Exception e) {
                log.error("回放消息失败", e);
                // Failed messages pushed back to queue tail
                redisTemplate.opsForList().rightPushAll(dumpKey, messages);
            }
        }
    }
}

3. Redis-Based Idempotency Implementation (Prevent Duplicate Consumption)

Key highlight: use Redis SETNX with expiry for efficient idempotency; performance >10x better than database unique index.

@Component
public class IdempotentService {
    @Autowired
    private StringRedisTemplate redisTemplate;

    // Highlight: Redis SETNX + expiry for high-performance idempotency
    // Performance 10x+ better than DB unique index
    public boolean isProcessed(String messageId) {
        String key = "mq:idempotent:" + messageId;
        // SETNX: only sets if key absent
        Boolean result = redisTemplate.opsForValue()
                .setIfAbsent(key, "1", Duration.ofHours(24));
        return Boolean.FALSE.equals(result);
    }
}

// Usage
if (idempotentService.isProcessed(msg.getMsgId())) {
    log.info("消息已处理,跳过: {}", msg.getMsgId());
    return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}

Technical Challenges & Solutions

Partition count vs. consume capacity conflict: In Kafka/RocketMQ, one partition = one consumer thread, so capacity capped by partition count. Solutions: 1) Emergency: degrade non-core + dump messages. 2) Long-term: plan partitions by peak TPS/1000. 3) Dynamic: support online partition scaling. Technical highlight: online partition scaling with ordering guarantee using "scale first, migrate later" strategy.

Message ordering vs. concurrent consumption conflict: Concurrency breaks order, but single-thread throughput too low. Solutions: 1) Local ordering: hash by business ID (e.g., order ID) to partition. 2) Segment lock: serialize same business ID messages. 3) Versioning: optimistic lock for eventual consistency. Technical highlight: Redis distributed lock for same business ID serial processing, balancing order and concurrency.

Message dump vs. no-loss guarantee: Dump process crash may lose messages. Solution: Two-phase commit: 1) write message to dump storage. 2) commit offset only after write succeeds. 3) if dump fails, don't commit offset; re-pull next time. Technical highlight: transactional messages or local transaction table to ensure atomicity of dump and offset commit.

Dead-letter automation: Unhandled dead letters accumulate, eventually fill disk. Solutions: 1) Configure DLQ, auto-forward messages failing >3 times. 2) Scheduled job auto-retry with exponential backoff. 3) Exceed max retries → manual handling. Technical highlight: exponential backoff: 1st retry 1 min, 2nd 5 min, 3rd 30 min.

High-throughput idempotency: Duplicate consumption causes data inconsistency; DB unique index performs poorly. Solutions: 1) Prefer Redis SETNX. 2) DB unique index as fallback. 3) Design naturally idempotent business operations. Technical highlight: dual idempotency check on message ID + business ID, balancing performance and reliability.

Consumer slow SQL avalanche: One slow SQL blocks entire consumer thread pool, halting all messages. Solutions: 1) Set timeouts on all DB operations. 2) Use Hystrix/Sentinel circuit breaker for slow SQL. 3) Isolate DB operations in separate thread pool. Technical highlight: thread pool isolation: core and non-core business use different thread pools.

Phase 3: Long-Term Prevention Measures

1. Producer Traffic Control

Integrate Sentinel/Hystrix rate limiting to cap producer send rate.

Smooth batch jobs (peak shaving) to avoid sudden million-message bursts at midnight.

2. Consumer Performance Optimization

Batch consumption: pull 10-50 messages per batch to reduce network round-trips.

Async processing: offload non-core logic to separate thread pools.

Thread pool tuning: adjust core threads and queue size per business scenario.

3. Comprehensive Monitoring & Alerting

Core metrics: message backlog, produce TPS, consume TPS, consume latency.

Alert thresholds: backlog >1,000 → warning; >10,000 → critical phone alert.

Multi-channel alerts: SMS, Enterprise WeChat, phone calls.

4. Dead-Letter Queue Automation

Configure DLQ to auto-forward messages failing >3 times.

Scheduled job auto-retries dead letters; exceed max retries → manual handling.

5. Quarterly Disaster Drills

Simulate message backlog scenario each quarter to validate emergency flow.

Test scaling, degradation, dump operations for latency and effectiveness.

Interview Bonus Points

Accurately state partition-consumer thread relationship : one partition = one consumer thread, so max threads = partition count.

Mention message idempotency : during scaling and retries, must guarantee no duplicate consumption.

Mention message ordering : if business requires order, scaling consumers can't be arbitrary.

Mention canary release : route 10% traffic to new consumer version first to avoid full-scale backlog.

Mention MQ cluster capacity planning : pre-plan partitions and brokers based on business growth to avoid emergency scaling.

Interviewer High-Frequency Follow-Up Questions

Follow-up 1: If scaling to partition limit still backlogged?

Answer: First degrade non-core business, then temporarily dump messages, later optimize consume logic or increase partitions.

Follow-up 2: How to ensure no message loss during dump?

Answer: Use "write dump success first, then commit offset" approach, ensuring at-least-once processing.

Follow-up 3: How to avoid dead-letter generation?

Answer: Implement idempotency, set reasonable retry counts, mark unrecoverable exceptions as dead-letter immediately.

Interview Transcript: Real-World Scenario

Setting the Scene

Interviewer: Let's discuss a scenario design question — production MQ message backlog, how would you handle it? Combine with your practical experience.

Candidate: Sure. During an e-commerce flash sale, RocketMQ suddenly piled up 8 million messages, monitoring all red. I followed the "stop bleeding → investigate → optimize" nine-character mantra.

Mitigation Details

Interviewer: Explain the stop-bleeding steps.

Candidate: First, emergency consumer scaling . If consumer group not saturated, add machines directly. In RocketMQ, as long as instance count ≤ read queue count, each added machine linearly boosts consume capacity. We had 16 read queues but only 4 consumers; scaled to 16 instances, consume speed doubled instantly.

Interviewer: What if read queues are already maxed out, can't add consumers?

Candidate: Depends on scenario. If brief disorder acceptable, dynamically increase read/write queues then add consumers. But for strict ordering like orders, I create an emergency "flood discharge topic", move backlog messages via script, then throw 20 temporary machines at it, fully isolated from original business — industry calls this temporary queue flood discharge .

Interviewer: Besides scaling, any other mitigation tactics?

Candidate: Yes, business degradation + discard non-critical messages . I flipped a feature flag, dropped telemetry logs and user behavior messages by directly returning return and recording offset, concentrating resources on orders and payments. Core backlog started draining within 10 minutes.

Root Cause Investigation

Interviewer: Good, bleeding stopped. How did you investigate root cause to prevent recurrence?

Candidate: Right, 90% of backlogs are slow consumption. I used Arthas trace to capture consume latency distribution; found one message taking 3 seconds, all stuck on downstream coupon RPC, P99 timeout severe.

Parallel investigation of common bottlenecks:

Slow SQL / missing index: threads stuck in JDBC. Tools: slow query log, Arthas trace.

Downstream RPC timeout: massive TimeoutException. Tools: SkyWalking, call chain stats.

Frequent GC: intermittent consume pauses. Tools: jstat -gc, GC logs.

Lock contention: threads piled in synchronized. Tools: jstack or Arthas thread -b.

Serial message processing: single consume latency extremely high. Check: consumer pull parameters.

Interviewer: How did you fix that coupon interface?

Candidate: Added local cache + degradation fallback upstream; when interface down, serve from cache or default coupon. Single message processing dropped from 3s to 20ms, backlog vanished rapidly.

Long-Term Optimizations

Interviewer: What long-term improvements did you make?

Candidate: Several lasting measures:

Batch consumption: tuned consumeMessageBatchMaxSize to reduce network round-trips.

Increase concurrency: adjusted consumeThreadMin/Max to raise thread pool.

Async DB write: changed post-consume DB write to async + queue, not blocking main consume flow.

Monitoring & rate limiting: producer rate limiting, real-time backlog alerts, early detection.

Interviewer: Any pitfalls with ordered messages?

Candidate: Ordered message scaling needs extreme care. Adding read/write queues directly routes same key to different queues, breaking order. Usually pause non-ordered consumption , shift all resources to ordered queues, or horizontally split by business key, increase partitions, but guarantee single key always lands in same queue.

Interviewer: Any other hard-earned lessons?

Candidate: Load test the consume link before big promotions . Many teams only stress-test producer; consume link gets dragged down, one rollover = major incident. We nearly missed Double 11 once; now every stress test includes consume stage.

Core Code Highlights from the Interview

Highlight ① Batch Consumption + Dynamic Thread Pool

RocketMQ defaults to single-message pull; we switched to batch consumption and made thread pool parameters dynamically adjustable without restart to handle sudden backlogs.

// Consumer core config
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("order_group");
consumer.setNamesrvAddr("namesrv:9876");
consumer.setConsumeMessageBatchMaxSize(64);         // pull 64 at once
consumer.setPullBatchSize(32);                       // long-poll batch size

// Dynamic thread pool, supports runtime adjustment
ThreadPoolExecutor consumeExecutor = new ThreadPoolExecutor(
    20, 80, 60L, TimeUnit.SECONDS,
    new LinkedBlockingQueue<>(1000),
    new ThreadFactoryBuilder().setNameFormat("consume-%d").build(),
    new ThreadPoolExecutor.CallerRunsPolicy()         // backpressure: caller runs when queue full, never lose messages
);

consumer.registerMessageListener((MessageListenerConcurrently) (msgs, context) -> {
    // Thread pool async processing, main thread quickly returns CONSUME_SUCCESS placeholder
    consumeExecutor.submit(() -> {
        processBatch(msgs);                          // batch processing
    });
    return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
});

Technical highlight: CallerRunsPolicy implements backpressure — when thread pool saturated, the pull thread executes the task itself, rather consume slower than lose messages . Combined with Apollo dynamic config for core thread count, one command boosts throughput during flash sales.

Highlight ② Degradation Switch + Safe Offset Discard

When discarding non-core messages urgently, must guarantee no offset loss, otherwise restart would re-consume massive useless messages.

@Value("${degrade.switch:false}")
private boolean degradeSwitch;

@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
        ConsumeConcurrentlyContext context) {
    if (degradeSwitch) {
        // Only process order/payment core topics; others ack + log
        for (MessageExt msg : msgs) {
            if (!isCoreTopic(msg.getTopic())) {
                log.warn("降级丢弃非核心消息, msgId={}, offset={}",
                        msg.getMsgId(), msg.getQueueOffset());
                // Still return SUCCESS so broker acknowledges, no redelivery
            } else {
                processCore(msg);
            }
        }
        return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
    }
    // Normal consume logic...
}

Technical highlight: Non-core messages directly return CONSUME_SUCCESS, releasing backlog while avoiding redelivery . Offset logged for later replay. Switch lives in config center, seconds-level effect .

Highlight ③ Async DB Write + Local Queue Buffer

Post-consume DB write is the biggest bottleneck; we used in-memory queue async batch write to decouple consume and persistence.

// Batch write task
private final LinkedBlockingQueue<OrderEvent> buffer = new LinkedBlockingQueue<>(10000);

@PostConstruct
public void init() {
    // Single thread batch flush, trigger every 200ms or 100 items
    ScheduledExecutorService flusher = Executors.newSingleThreadScheduledExecutor();
    flusher.scheduleAtFixedRate(() -> {
        List<OrderEvent> batch = new ArrayList<>();
        buffer.drainTo(batch, 100);
        if (!batch.isEmpty()) {
            orderDao.batchInsert(batch);
        }
    }, 0, 200, TimeUnit.MILLISECONDS);
}

public void processBatch(List<MessageExt> msgs) {
    for (MessageExt msg : msgs) {
        OrderEvent event = parse(msg);
        if (!buffer.offer(event)) {                 // queue full
            orderDao.insert(event);                 // degrade to sync write, backpressure to consume
        }
    }
}

Technical highlight: LinkedBlockingQueue + scheduled batch flush, strips disk IO from main consume path . Queue full → auto degrade to sync write, ensuring no data loss while limiting memory usage.

Technical Challenges Summary

Fast flood discharge without message loss: Consumers at limit, can't add machines; direct queue scaling may break order. Solution: Temporary topic flood discharge + script transfer; non-ordered messages dynamic queue scaling.

Ordered message scaling: Adding write queues routes single key to multiple queues, causing disorder. Solution: Pause non-ordered consumption, horizontally split partitions by business key, guarantee single key always same queue.

Consume bottleneck localization: Symptom is backlog; root cause could be slow SQL, RPC timeout, GC, lock contention. Solution: Full-chain tracing + Arthas trace to pinpoint latency hotspots.

Degradation vs. duplicate consumption trade-off: Discarding non-core messages leads to useless re-consumption on retry. Solution: Return CONSUME_SUCCESS directly to ack, log offset for later replay.

Scaling smoothness: Changing queue count may require consumer restart, causing brief consume pause. Solution: Leverage RocketMQ dynamic queue add/remove, client unaware; combined with rolling restart.

Producer cascade avalanche: Slow consume fills broker, blocks producers. Solution: Producer async send + rate limiting to protect broker write; consumer fast degradation.

Consumer thread pool tuning: Too few threads = slow consume; too many = OOM or context-switch overhead. Solution: Dynamic thread pool + load-test baseline, with CallerRunsPolicy for backpressure.

Post-incident data补全: Degraded-discarded messages need business data补全. Solution: Log key and offset at consume time; offline script batch replay.

Interviewer: Well summarized, especially the dynamic thread pool and async DB write — achieving higher throughput without losing messages is indeed the core solution for high-concurrency scenarios.

Candidate: Thanks. These are lessons paid for with blood and sweat during several flash-sale incidents. Each post-mortem yields a reusable component; now new team members can use them from day one.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Performance OptimizationMQRocketMQIdempotencyDead Letter QueueBatch ConsumptionMessage BacklogProduction Incident Response
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.