How to Prevent Duplicate Message Consumption in RabbitMQ

In RabbitMQ, duplicate message consumption is a common issue that occurs when consumers experience network fluctuations, node failures, or ACK failures, causing the broker to re‑deliver messages.

Message Idempotency Handling (Business Side)

Idempotency means that multiple executions of the same operation have the same effect as a single execution; in message processing this ensures the business impact remains consistent regardless of how many times a message is consumed.

The consumer can record identifiers of processed messages, for example by maintaining an in‑memory Set and checking the message ID before processing; if the ID already exists, the message is ignored, otherwise it is processed and the ID is added.

public class MessageProcessor {
    // 已消费的消息
    private static final Set<String> processedMessages = new HashSet<>();

    public void processMessage(String messageId, String messageContent) {
        if (processedMessages.contains(messageId)) {
            // 消息已处理过，直接返回
            return;
        }
        // 处理消息
        System.out.println("处理消息：" + messageContent);
        // 将消息标识添加到已处理集合
        processedMessages.add(messageId);
    }
}

Use a cache such as Redis to store processed message identifiers, checking the cache before handling a message and optionally setting an expiration to avoid unbounded growth.

Alternatively, maintain a deduplication table in a database that records unique keys of processed messages and query it before processing.

Using RabbitMQ’s Acknowledgment Mechanism

RabbitMQ provides automatic acknowledgment (autoAck=true) and manual acknowledgment (autoAck=false). Automatic mode may cause duplicate consumption because the broker assumes the message is processed as soon as it is delivered. With manual acknowledgment, the consumer must explicitly call basicAck after successful processing; if the consumer crashes or fails to send basicAck, the broker will re‑queue the message.

public class ManualAckConsumer {
    private static final String QUEUE_NAME = "normal_ack_queue";

    public static void main(String[] argv) throws Exception {
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("localhost");
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();

        channel.queueDeclare(QUEUE_NAME, false, false, false, null);
        boolean autoAck = false;
        channel.basicConsume(QUEUE_NAME, autoAck, "normal-ack-consumer",
            false, false, null, new DefaultConsumer(channel) {
                @Override
                public void handleDelivery(String consumerTag,
                                           Envelope envelope,
                                           AMQP.BasicProperties properties,
                                           byte[] body) throws IOException {
                    String message = new String(body, "UTF-8");
                    System.out.println("[x] Received: '" + message + "'");
                    try {
                        // Simulate processing
                        Thread.sleep(1000);
                        // Manual ack
                        channel.basicAck(envelope.getDeliveryTag(), false);
                        System.out.println("[x] Message acked");
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                }
            });
    }
}

Using RabbitMQ’s Message Deduplication Plugin

By adding a unique ID to message properties, the Message Deduplication plugin can deduplicate messages on the server side. It maintains a deduplication table (in memory or on disk) of processed message IDs; when a new message arrives, the plugin checks the table and discards the message if the ID already exists, otherwise it records the ID and processes the message normally.