Kafka 4.2 Share Groups: Can Kafka Replace RabbitMQ for Task Queues?

Our project has long used both Kafka and RabbitMQ: Kafka for event streams (orders, payments, tracking) and RabbitMQ for background tasks like report generation, image processing, and bulk exports. This division made sense because Kafka excels at event streaming while RabbitMQ excels at task queues. However, Kafka 4.2 introduces Share Groups (also called Kafka Queues), which promise queue-like semantics. The author evaluates whether Kafka can now handle traditional task-queue workloads by migrating a report-generation queue.

Why Traditional Kafka Struggles with Task Queues

With a traditional Kafka consumer group, each partition is assigned to at most one consumer. If a topic has only 3 partitions, launching 10 consumers leaves 7 idle. For event streams this is desirable (preserves per-partition order), but for independent tasks like "generate Zhang's monthly report" and "generate Li's monthly report" no ordering is needed. The old workaround was to over-partition (e.g., 30 or 100 partitions) just to allow more consumers, which is wasteful.

Share Groups Change the Consumption Model

Kafka 4.2 Share Groups allow multiple consumers in the same group to read different records from the same partition. The broker distributes records among consumers, so consumer count can exceed partition count. The trade-off: per-partition ordering is lost, which is acceptable for independent tasks.

Minimal Spring Boot Implementation

Stack used: Java 21, Spring Boot 4.1.1, Spring Kafka 4.1.1, Apache Kafka 4.2.x. The producer remains unchanged; only the consumer side switches to Share Group APIs.

Dependencies (pom.xml)

org.springframework.boot
spring-boot-starter-parent
4.1.1
21

org.springframework.boot
spring-boot-starter-web

org.springframework.boot
spring-boot-starter-kafka

Producer Configuration

spring:
  kafka:
    bootstrap-servers: 127.0.0.1:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer

Task Record

public record ReportTask(
    String taskId,
    Long userId,
    String reportType) {}

REST Endpoint to Submit Tasks

@RestController
@RequestMapping("/reports")
public class ReportController {
    private final KafkaTemplate<String, String> kafkaTemplate;
    private final ObjectMapper objectMapper;
    public ReportController(KafkaTemplate<String, String> kafkaTemplate, ObjectMapper objectMapper) {
        this.kafkaTemplate = kafkaTemplate;
        this.objectMapper = objectMapper;
    }
    @PostMapping
    public void submit(@RequestBody ReportTask task) throws Exception {
        String json = objectMapper.writeValueAsString(task);
        kafkaTemplate.send("report-generate", task.taskId(), json);
    }
}

Share Consumer Configuration

Instead of ConsumerFactory and ConcurrentKafkaListenerContainerFactory, use ShareConsumerFactory and ShareKafkaListenerContainerFactory:

@Configuration
@EnableKafka
public class ReportConsumerConfig {
    @Bean
    ShareConsumerFactory<String, String> shareConsumerFactory() {
        Map<String, Object> properties = new HashMap<>();
        properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
        properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        return new DefaultShareConsumerFactory<>(properties);
    }
    @Bean
    ShareKafkaListenerContainerFactory<String, String> shareKafkaListenerContainerFactory(
            ShareConsumerFactory<String, String> shareConsumerFactory) {
        ShareKafkaListenerContainerFactory<String, String> factory =
                new ShareKafkaListenerContainerFactory<>(shareConsumerFactory);
        factory.setConcurrency(5);
        return factory;
    }
}

Consumer with Manual Acknowledgment

Spring Kafka 4.1 integrates Share Consumer into the familiar @KafkaListener model. To distinguish failure types, set ShareAckMode.MANUAL:

@Bean
ShareKafkaListenerContainerFactory<String, String> manualShareKafkaListenerContainerFactory(
        ShareConsumerFactory<String, String> shareConsumerFactory) {
    ShareKafkaListenerContainerFactory<String, String> factory =
            new ShareKafkaListenerContainerFactory<>(shareConsumerFactory);
    factory.setConcurrency(5);
    factory.getContainerProperties().setShareAckMode(ContainerProperties.ShareAckMode.MANUAL);
    return factory;
}

Consumer code:

@Component
public class ReportTaskConsumer {
    private final ObjectMapper objectMapper;
    private final ReportService reportService;
    public ReportTaskConsumer(ObjectMapper objectMapper, ReportService reportService) {
        this.objectMapper = objectMapper;
        this.reportService = reportService;
    }
    @KafkaListener(
        topics = "report-generate",
        groupId = "report-workers",
        containerFactory = "manualShareKafkaListenerContainerFactory")
    public void consume(ConsumerRecord<String, String> record,
                        ShareAcknowledgment acknowledgment) throws Exception {
        ReportTask task = objectMapper.readValue(record.value(), ReportTask.class);
        try {
            reportService.generate(task);
            acknowledgment.acknowledge(); // ACCEPT
        } catch (TemporaryReportException e) {
            acknowledgment.release(); // RELEASE: retry later
        } catch (IllegalReportRequestException e) {
            acknowledgment.reject(); // REJECT: permanent failure
        }
    }
}

ACK Semantics: ACCEPT, RELEASE, REJECT, RENEW

ACCEPT (acknowledge): task succeeded, remove record.

RELEASE (release): transient failure, re-queue for redelivery.

REJECT (reject): permanent failure, discard (or send to DLQ).

RENEW (renew): extend the acquisition lock while processing continues.

Kafka 4.2 tracks a broker-side delivery count (default max 5 attempts) to prevent poison-pill loops. After the limit, the record enters an Archived state. The application does not see an exact retry count, so fine-grained retry policies still require custom tracking.

Handling Long-Running Tasks with RENEW

The default acquisition lock duration ( share.record.lock.duration.ms) is 30 seconds. If a PDF report takes 2 minutes, the lock may expire and another consumer could pick up the same record, causing duplicate work. acknowledgment.renew() extends the lock. Example pattern:

try {
    reportService.prepareData(task);
    acknowledgment.renew();
    reportService.renderPdf(task);
    acknowledgment.renew();
    reportService.upload(task);
    acknowledgment.acknowledge();
} catch (TemporaryReportException e) {
    acknowledgment.release();
} catch (Exception e) {
    acknowledgment.reject();
}

renew()

is not a final acknowledgment; the record must eventually receive ACCEPT, RELEASE, or REJECT.

Single-Broker Development Pitfall

Share Groups use an internal topic __share_group_state with a default replication factor of 3. On a single-broker dev cluster, topic creation fails. Set in broker config:

share.coordinator.state.topic.replication.factor=1
share.coordinator.state.topic.min.isr=1

Production clusters with three or more brokers typically need no change. The author also recommends Kafka 4.2.1 over 4.2.0 due to a critical Share Group deadlock fix.

Can We Delete RabbitMQ?

The author concludes no. Share Groups solve a specific pain point: high-volume independent tasks where consumer elasticity matters and per-partition ordering is irrelevant. Good candidates include image compression, video transcoding, PDF/Excel generation, email/SMS batches, AI inference, file parsing, and data cleansing. If the organization already runs Kafka, adding Share Groups for these tasks can reduce operational overhead.

However, RabbitMQ retains mature routing (exchanges, routing keys), TTL, priority queues, and dead-letter exchanges. For complex routing or teams deeply invested in RabbitMQ, forced migration for "stack unification" is unjustified. Ordering-sensitive flows (e.g., order state transitions CREATED → PAID → SHIPPED → FINISHED) should stay on traditional Kafka consumer groups with key-based partitioning.

Kafka 4.2 does not replace the old consumer group; it adds a parallel consumption mode. Use classic Kafka for event streams and ordering; use Share Groups for elastic task dispatch. The old adage "Kafka is a log, RabbitMQ is a queue" now needs a footnote: Kafka 4.2 can seriously ask, "Can I handle this task queue too?"