MQ Production Failure Troubleshooting: Message Loss, Duplicates, Backlog & Dead Letters

Five Core MQ Production Failures

All MQ production anomalies across RocketMQ, Kafka, and RabbitMQ fall into five categories: message loss, duplicate consumption, massive message backlog, consumer thread hangs, and dead letter queue blocking. These are the focus of enterprise interviews and incident retrospectives.

Message Loss (P0 Severity)

Message loss is the most severe MQ incident, causing irreversible business data loss. It occurs in three phases:

1. Producer Send Phase (Most Frequent)

Root cause: No send result verification, no reliable delivery, no retry mechanism, no logging. Network jitter causes silent failures.

Typical scenarios: Async send without callback, ignored send exceptions, uncaught send failures.

Fixes:

Prefer synchronous or callback-based async send; always verify send result.

Enable send retry for transient network jitter.

Persist messages to local log or local transaction table before send for traceability and replay.

Route failed sends to a compensation queue with scheduled retry.

2. Broker Storage Phase

Root cause: Persistence disabled, master-slave sync incomplete, cluster crash/restart, improper flush policy.

Fixes:

Enforce disk persistence in production; prohibit in-memory-only storage.

Require master-slave sync success before acknowledging send.

Tune flush policy: synchronous flush for core business, async for ordinary to balance performance and reliability.

3. Consumer Consumption Phase

Root cause: Business exception or crash before ack; exception caught but not triggering retry, causing silent drop.

Fixes:

Business exceptions must return consume retry ; never ack success on failure.

Global exception interceptor to catch unknown errors and trigger retry.

Distinguish ignorable business exceptions from retry-worthy ones for precise control.

Duplicate Consumption (Most Stubborn)

MQs guarantee at-least-once delivery, not exactly-once. Network retries, cluster retries, ack timeouts cause redelivery. Duplicates cause dirty data: duplicate orders, double deductions, duplicate points, duplicate ledger entries.

Core Triggers

Consumer processes successfully but ack times out; broker redelivers.

Producer retry sends duplicate messages.

Cluster failover or node restart triggers redelivery.

Batch consumption partial failure triggers full batch retry.

Only Production Fix: Consumer Idempotency

Goal: make duplicate consumption harmless. Three patterns by priority:

Unique primary key idempotency (best): Use message unique ID or business order number as unique key; DB unique index blocks duplicate inserts.

Local state table idempotency: Check consumption record before processing; skip if exists, else process and record.

Distributed lock idempotency: Acquire lock before consumption; ensures only one consumer processes a given message at a time.

Massive Message Backlog (Most Frequent)

Symptom: growing queue depth, rising task latency, delayed scheduled tasks, async chain timeouts. Core essence: consumption speed < production speed .

Four Root Causes

Slow consumption logic: Slow SQL, remote calls, sync I/O, heavy loops per message.

Insufficient consumer concurrency: Small thread pool, too few consumer nodes.

Consumption blocking/hang: Infinite loops, external API timeout without release, lock waits.

Exception messages retry endlessly: Single failed message retries infinitely, blocking subsequent messages.

Emergency Mitigation (Fast Recovery)

Temporarily scale out consumer nodes to boost concurrency.

Temporarily increase consumer thread count.

Identify stuck poison messages; manually skip or move to DLQ.

Pause non-core message production; prioritize draining backlog.

Long-term Optimization

Offload time-consuming logic from consumer; async non-core flows.

Optimize slow SQL and third-party calls to reduce per-message latency.

Set sensible retry limits; auto-route failures to DLQ to avoid infinite blocking.

Split queues by business; isolate core traffic from non-core.

Consumer Hang / Thread Stuck (Most Deceptive)

Service alive, MQ connection healthy, no error logs, threads not exiting — yet zero consumption, queue keeps growing.

Core Causes

External calls without timeout config → threads block forever.

Code infinite loops, deadlocks.

Batch consumption hits exception but doesn't exit; batch stalls.

Local resources not released; threads blocked.

Diagnosis & Fix

Thread stack analysis to pinpoint blocking code line.

Enforce timeouts on all external calls; eliminate indefinite blocking.

Add consumption monitoring, latency metrics, exception logging for fast detection.

Configure thread timeout auto-release and auto-retry on exception.

Dead Letter Queue Blocking

DLQ is the safety net: messages failing repeated consumption move to DLQ and exit normal flow. Teams often ignore DLQ, causing massive abnormal message buildup, consuming cluster resources, degrading normal traffic.

Three Entry Conditions

Retry attempts exhausted, still failing.

Message expires unconsumed.

Queue length exceeded; broker actively discards to DLQ.

Governance Rules

DLQ must alert: Immediate warning on any DLQ data; zero tolerance for accumulation.

Regular root-cause review: Analyze failure reasons, fix code bugs.

Support manual replay: One-click retry or batch replay for DLQ messages to recover business data.

Periodic cleanup/archive: Scheduled purge of invalid DLQ data to free resources.

Message Ordering Violations

Order status transitions, payment flows, inventory changes require strict ordering ; out-of-order causes state machine errors.

Causes

Multiple consumers concurrent; later message finishes first.

Retry mechanism delays old message, breaking sequence.

Multiple queues mixed; business messages scattered.

Ordering Solutions

Dedicated queue for ordered business; no multi-business mixing.

Ensure same business ID routes to same partition/queue.

Disable concurrent consumption for ordered topics; use single-threaded serial consumption.

Custom sequence validation before consumption.

Universal MQ Incident SOP (Reusable)

Metric observation: Produce QPS, consume QPS, backlog, retry count, DLQ size, consume latency.

Log investigation: Search consume errors, stack traces, send failures; locate abnormal messages.

Thread analysis: Check consumer threads for block, hang, stuck.

Message validation: Verify loss, duplication, ordering, parameter anomalies.

Emergency stop-gap: Scale concurrency, clear stuck messages, temporary degrade, replay missing messages.

Root-cause fix: Patch code bugs, optimize consume logic, add idempotency,完善 retry & DLQ mechanisms.

Monitoring safety net: Add backlog alerts, DLQ alerts, latency alerts, message loss warnings.

Summary

This article delivers complete coverage of MQ production failures across six core problems. You gain: rapid mitigation, precise root-cause isolation, code-level fixes, and architectural guardrails — fully closing the gap in middleware troubleshooting capability.

Next Episode Preview

Next: Nginx/Gateway layer production troubleshooting — tackling gateway timeouts, load imbalance, rate-limit/circuit-breaker anomalies, CORS failures, reverse proxy failures, traffic jitter — completing the middleware series.