Comprehensive Guide to Spring‑Kafka: Integration, Configuration, and Advanced Features

Kafka is a high‑performance message queue built on topics and partitions, and Spring‑Kafka wraps the Apache Kafka client to simplify integration in Spring applications. The article begins by showing the Maven dependency required for Spring‑Kafka and a minimal Spring Boot configuration that sets the bootstrap server address.

Simple integration is demonstrated with a @SpringBootApplication class that defines a KafkaTemplate bean, a REST endpoint @GetMapping("/send/{input}") that sends a message, and a @KafkaListener method that logs the received value.

For testing, the article introduces @EmbeddedKafka, which starts an in‑memory Kafka broker with configurable broker count, ports, and properties, allowing developers to run unit tests without an external Kafka cluster.

Topic creation is covered in three ways: using Spring's KafkaAdmin bean with NewTopic, using the low‑level AdminClient API, and invoking the Kafka command‑line tool TopicCommand. The article also shows how to create topics with the older kafka_2.10 client.

Message sending with KafkaTemplate is explored in both asynchronous and synchronous modes. Asynchronous sending uses addCallback to handle success and failure, while synchronous sending retrieves the SendResult via Future.get(). Transactional messaging is enabled by setting spring.kafka.producer.transaction-id-prefix and wrapping sends in template.executeInTransaction(...) or annotating methods with @Transactional.

The request‑reply pattern is implemented with ReplyingKafkaTemplate. A bean creates a reply container, the template is wired, and a REST endpoint sends a ProducerRecord and waits for the reply, demonstrating RPC‑style communication over Kafka.

Advanced @KafkaListener usage includes specifying topic partitions, initial offsets, concurrency, and a custom error handler bean ( KafkaListenerErrorHandler). Manual acknowledgment mode is shown by disabling auto‑commit and adding an Acknowledgment parameter, allowing the listener to decide when to commit offsets.

Listener lifecycle control is achieved with KafkaListenerEndpointRegistry, which can start, pause, or resume listeners identified by their id. The article provides REST endpoints that invoke these actions.

Message forwarding with @SendTo is illustrated, allowing a listener to forward processed data to another topic. Finally, retry and dead‑letter handling are configured using a SeekToCurrentErrorHandler with a DeadLetterPublishingRecoverer, retrying up to three times before routing failed messages to a .DLT topic.

The conclusion reflects on the usefulness of Spring‑Kafka’s rich feature set, including embedded brokers, request‑reply semantics, and transactional support, encouraging developers to adopt these patterns to avoid common pitfalls.