Master Kafka on Linux: Install, Visualize with Kafka‑Eagle, and Use Spring Boot

Kafka is a popular messaging middleware used by thousands of companies. This article covers Kafka installation on Linux, a visual management tool, and integration with Spring Boot to help you get started quickly.

Kafka Introduction

Kafka is an open‑source distributed streaming platform developed by LinkedIn, written in Scala and Java. It provides a high‑throughput, low‑latency platform for real‑time data processing based on a publish‑subscribe model.

Key features of Kafka include:

High throughput, low latency (as low as 2 ms).

Scalability to thousands of brokers and millions of partitions.

Durable storage with fault‑tolerant clusters.

High availability across zones.

Kafka Installation

We install Kafka directly on CentOS 7.6 without Docker.

Download the package: kafka_2.13-2.8.0.tgz

Extract the package to the desired directory:

cd /mydata/kafka/
 tar -xzf kafka_2.13-2.8.0.tgz

Enter the extracted directory:

cd kafka_2.13-2.8.0

Start Zookeeper (default port 2181):

# Run in background and log to zookeeper-out.file
nohup bin/zookeeper-server-start.sh config/zookeeper.properties > zookeeper-out.file 2>&1 &

Modify config/server.properties to set the listener address (e.g., PLAINTEXT://192.168.5.78:9092).

# Socket Server Settings
listeners=PLAINTEXT://192.168.5.78:9092

Start Kafka (default port 9092):

# Run in background and log to kafka-out.file
nohup bin/kafka-server-start.sh config/server.properties > kafka-out.file 2>&1 &

Kafka Command‑Line Operations

Use the CLI to create topics, send and consume messages.

Create a topic named consoleTopic:

bin/kafka-topics.sh --create --topic consoleTopic --bootstrap-server 192.168.5.78:9092

Describe the topic:

bin/kafka-topics.sh --describe --topic consoleTopic --bootstrap-server 192.168.5.78:9092

Send messages:

bin/kafka-console-producer.sh --topic consoleTopic --bootstrap-server 192.168.5.78:9092

Consume messages from the beginning:

bin/kafka-console-consumer.sh --topic consoleTopic --from-beginning --bootstrap-server 192.168.5.78:9092

Kafka Visualization with Kafka‑Eagle

Command‑line operations can be cumbersome, so we use the visual tool kafka‑eagle .

Install JDK

CentOS does not include a full JDK by default.

Download JDK 8 from AdoptOpenJDK and extract it.

cd /mydata/java
 tar -zxvf OpenJDK8U-jdk_x64_linux_xxx.tar.gz
 mv OpenJDK8U-jdk_x64_linux_xxx.tar.gz jdk1.8

Add JAVA_HOME to /etc/profile and reload:

vi /etc/profile
# Add to profile
export JAVA_HOME=/mydata/java/jdk1.8
export PATH=$PATH:$JAVA_HOME/bin
. /etc/profile

Install kafka‑eagle

Download the binary from GitHub releases and extract it.

cd /mydata/kafka/
 tar -zxvf kafka-eagle-web-2.0.5-bin.tar.gz

Add KE_HOME to /etc/profile:

vi /etc/profile
# Add to profile
export KE_HOME=/mydata/kafka/kafka-eagle-web-2.0.5
export PATH=$PATH:$KE_HOME/bin
. /etc/profile

Configure MySQL in $KE_HOME/conf/system-config.properties (set driver, URL, username, password).

# kafka mysql jdbc driver address
kafka.eagle.driver=com.mysql.cj.jdbc.Driver
kafka.eagle.url=jdbc:mysql://localhost:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
kafka.eagle.username=root
kafka.eagle.password=root

Start kafka‑eagle:

$KE_HOME/bin/ke.sh start

Useful commands: stop, restart, status, stats, and view logs with tail -f $KE_HOME/logs/ke_console.out.

Access the UI at http://192.168.5.78:8048/ (login: admin:123456).

Using the Visual Tool

Create topics, send messages, and monitor clusters directly from the UI.

Consume messages via the UI or use the built‑in KSQL console for SQL‑style queries.

Spring Boot Integration

Integrating Kafka with Spring Boot is straightforward.

Add the Spring Kafka dependency to pom.xml:

<!--Spring integration with Kafka-->
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>2.7.1</version>
</dependency>

Configure application.yml with bootstrap servers and consumer group ID:

server:
  port: 8088
spring:
  kafka:
    bootstrap-servers: '192.168.5.78:9092'
    consumer:
      group-id: "bootGroup"

Create a producer component to send messages:

@Component
public class KafkaProducer {
    @Autowired
    private KafkaTemplate kafkaTemplate;

    public void send(String message) {
        kafkaTemplate.send("bootTopic", message);
    }
}

Create a consumer component to receive messages:

@Slf4j
@Component
public class KafkaConsumer {
    @KafkaListener(topics = "bootTopic")
    public void processMessage(String content) {
        log.info("consumer processMessage : {}", content);
    }
}

Expose a REST endpoint to trigger message sending:

@Api(tags = "KafkaController", description = "Kafka functionality test")
@Controller
@RequestMapping("/kafka")
public class KafkaController {
    @Autowired
    private KafkaProducer kafkaProducer;

    @ApiOperation("Send message")
    @RequestMapping(value = "/sendMessage", method = RequestMethod.GET)
    @ResponseBody
    public CommonResult sendMessage(@RequestParam String message) {
        kafkaProducer.send(message);
        return CommonResult.success(null);
    }
}

Console output confirms the message was received and processed:

2021-05-19 16:59:21.016  INFO 2344 --- [ntainer#0-0-C-1] c.m.mall.tiny.component.KafkaConsumer : consumer processMessage : Spring Boot message!

Conclusion

By following this hands‑on guide, you should now be able to install Kafka, use the Kafka‑Eagle visual tool, and integrate Kafka with a Spring Boot application—essential steps for mastering this popular messaging middleware.