How to Guarantee No Message Loss in MQ Systems – Interview Guide

Preface

Message queues (MQ) are a high‑frequency interview topic; this article shares common MQ questions and how to answer them effectively.

Case Background

Using JD.com as an example, when a user purchases a product and uses JD beans for a discount, the transaction service sends a message like “deduct 100 JD beans from account X” to an MQ queue, and the JD‑bean service consumes the message to perform the actual deduction.

Case Analysis

The primary reasons for introducing MQ are system decoupling, traffic control, high availability, and high performance.

System decoupling: MQ isolates upstream and downstream services, allowing independent evolution and graceful degradation.

Traffic control: MQ can smooth traffic spikes (e.g., flash sales) by buffering requests according to downstream processing capacity.

However, MQ also introduces consistency challenges; ensuring that messages are not lost is a key concern.

Case Solution

A message passes through three stages: production, storage, and consumption.

Message production stage: As long as the producer receives an ACK from the broker, the send is considered successful; handling return values and exceptions prevents loss at this stage.

Message storage stage: The broker typically replicates messages to multiple nodes (e.g., at least two) before acknowledging, ensuring durability.

Message consumption stage: The consumer should acknowledge only after business logic succeeds, guaranteeing that processed messages are not lost.

Even with these safeguards, failures can still occur. Following the "Design for Failure" principle, a mechanism to detect lost messages is needed.

One approach is to assign each message a globally unique ID or an incrementing version number at the producer, then verify continuity on the consumer side.

Implementation can use an interceptor: the producer injects the ID/version into the message; the consumer checks continuity or consumption status, allowing detection without polluting business code.

If multiple producers/consumers exist, a globally unique ID is preferred over simple version numbers.

Beyond loss, interviewers often ask about duplicate consumption. The solution is to make the consumer idempotent, for example by recording processed message IDs in a log table (or using Redis) and checking before processing.

Handling message backlog involves scaling consumer instances and increasing topic partitions (e.g., in Kafka) to match consumer parallelism.

Summary

Identify where loss can occur (production, storage, consumption) and monitor each stage.

Ensure consumer idempotency to avoid duplicate processing.

Address backlog by expanding consumer capacity and partition count, and by monitoring performance.

When discussing these points, emphasize the thought process and design considerations rather than just the final solution.