How to Prevent Duplicate Message Consumption with Spring Cloud Stream Consumer Groups

Recently, many users have reported duplicate message consumption when using Spring Cloud Stream with RabbitMQ or Kafka. The root cause is often a misunderstanding of consumer groups, which were explained in earlier blog posts and the book Spring Cloud Microservices in Action .

In production, a service usually runs multiple instances that bind to the same topic. By default, each instance receives a copy of every message, leading to duplicate consumption. To ensure that only one instance processes a message, a consumer group must be configured.

Example: Using a Consumer Group

Step 1: Create a binding interface and bind the example-topic input channel.

interface ExampleBinder {
    String NAME = "example-topic";
    @Input(NAME)
    SubscribableChannel input();
}

Step 2: Implement the listener for the input channel.

@EnableBinding(ExampleBinder.class)
public class ExampleReceiver {
    private static final Logger logger = LoggerFactory.getLogger(ExampleReceiver.class);

    @StreamListener(ExampleBinder.NAME)
    public void receive(String payload) {
        logger.info("Received: " + payload);
    }
}

Step 3: Create the main application class and configuration.

@SpringBootApplication
public class ExampleApplication {
    public static void main(String[] args) {
        SpringApplication.run(ExampleApplication.class, args);
    }
}

# application.properties
spring.application.name=stream-consumer-group
server.port=0   # set to 0 to run multiple instances locally

Start two instances of the above application to reproduce the duplicate consumption scenario.

Building the Message Producer

Define an output binding with the same name so that messages are sent to the topic consumed by the instances.

@EnableBinding({ExampleApplicationTests.ExampleBinder.class})
public class ExampleApplicationTests {
    @Autowired
    private ExampleBinder exampleBinder;

    @Test
    public void exampleBinderTester() {
        exampleBinder.output().send(MessageBuilder.withPayload("Produce a message from : http://blog.didispace.com").build());
    }

    interface ExampleBinder {
        String NAME = "example-topic";
        @Output(NAME)
        MessageChannel output();
    }
}

Running the test shows that both instances receive the message and log Received: Produce a message from : http://blog.didispace.com, confirming duplicate consumption.

Resolving the Issue with Consumer Groups

Add the following configuration to assign a consumer group to the binding:

spring.cloud.stream.bindings.example-topic.group=aaa

When a consumer group is specified, only one subscriber in that group will receive each message, achieving load balancing and eliminating duplicate processing. The previous duplicate behavior occurred because, without an explicit group, each subscriber was assigned an anonymous group, resulting in a broadcast pattern.

Note that the example-topic name must match the name used in both @Input and @Output annotations.