Ensuring No Message Loss in MQ Systems: Interview Guide and Practical Solutions

Interviewers often ask candidates how to ensure that messages are never lost when using MQ technologies like Kafka, RabbitMQ, or RocketMQ; this question tests both theoretical knowledge and practical problem‑solving skills.

Using a JD e‑commerce example, the article shows how the transaction service sends a "deduct 100 beans" message to an MQ queue and the bean service consumes it to perform the actual deduction, illustrating a typical real‑world scenario.

The core motivations for introducing MQ are system decoupling and traffic control. Decoupling isolates upstream and downstream changes, while traffic shaping (e.g., spike‑shaving during flash sales) prevents overload, but both introduce consistency challenges that must be addressed.

Message loss can occur in three stages: production, storage, and consumption. Understanding each stage is essential for a complete answer.

Message production stage: As long as the producer receives an ACK from the broker and handles any exceptions, the message is considered successfully sent; no loss occurs here.

Message storage stage: The broker ensures durability by replicating messages to multiple nodes (typically at least two) before returning an ACK, guaranteeing persistence even if a single node fails.

Message consumption stage: Consumers should pull messages, process business logic, and only then send a commit/ACK to the broker. This guarantees that a message is not considered consumed until processing succeeds.

Because failures are inevitable in distributed systems, a "design‑for‑failure" approach is required: assign a globally unique ID or monotonically increasing version number to each message, and verify these on the consumer side to detect missing messages.

To prevent duplicate consumption, implement idempotent processing. A common pattern is a message‑log table (or a Redis set) storing message_id and its status; before processing, check if the ID already exists, ensuring exactly‑once execution.

Global unique ID generation can be achieved via database auto‑increment, UUID, Redis counters, or the Snowflake algorithm; each has trade‑offs between simplicity, performance, and availability, and the choice should align with business requirements.

When messages back‑log, the primary cause is insufficient consumer processing capacity. Solutions include scaling out consumer instances, temporarily degrading non‑critical services, and increasing the number of partitions for the topic (e.g., in Kafka) so that each consumer can handle a separate partition.

In summary, a complete interview answer should outline the three message stages, describe ACK and replication mechanisms, propose unique‑ID based loss detection, explain idempotent consumption via a log table, discuss ID generation options, and address back‑pressure handling through consumer scaling and partition management.