Deploy Kafka 3.8.1 KRaft Cluster in 10 Minutes with Docker Compose

Environment Preparation

Cluster Planning

Three servers with IPs: 10.11.18.13, 10.11.18.14, 10.11.18.15. Each runs one Kafka node; three nodes form the cluster.

Port Planning

9092 : Client traffic (produce/consume)

9093 : Controller internal communication

Note: If your server IPs differ, update the corresponding IPs in the configuration files.

Generate Cluster ID

KRaft mode requires a single cluster ID shared by all nodes. Generate once on any machine:

docker run --rm apache/kafka:3.8.1 /opt/kafka/bin/kafka-storage.sh random-uuid

Example output: MkU3OEVBNTcwNTJENDM2Qk. Record this ID for use on all three nodes.

Machine Initialization (Run on All Three Servers)

1. Create Data Directory

mkdir -p /data/kafka/data

2. Set Directory Permissions

Kafka container runs as UID 1000; grant read/write access:

chown -R 1000:1000 /data/kafka

3. Adjust Kernel Parameters

Kafka requires higher memory mapping and network connection limits:

cat >> /etc/sysctl.conf <<'EOF'
vm.max_map_count=262144
net.core.somaxconn=65535
net.ipv4.tcp_max_syn_backlog=65535
EOF
sysctl -p

4. Disable Swap

Kafka recommends disabling swap to avoid performance degradation from memory swapping. swapoff -a For persistence across reboots, comment out the swap mount entry in /etc/fstab.

Docker Compose Configuration

Node 1 (10.11.18.13)

Create /data/kafka/docker-compose.yml with the following content:

version: '3.8'

services:
  kafka:
    image: apache/kafka:3.8.1
    container_name: kafka-node-1
    restart: always
    network_mode: host
    user: "1000:1000"
    environment:
      # ===== Cluster Basics =====
      CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: [email protected]:9093,[email protected]:9093,[email protected]:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OPTS: "-Dcom.sun.management.jmxremote=false"
      # ===== Listeners =====
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://10.11.18.13:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      # ===== Storage =====
      KAFKA_LOG_DIRS: /opt/kafka/kraft-combined-logs
      # ===== Replication & Partitions =====
      KAFKA_NUM_PARTITIONS: 3
      KAFKA_DEFAULT_REPLICATION_FACTOR: 3
      KAFKA_MIN_INSYNC_REPLICAS: 2
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
      # ===== Log Retention =====
      KAFKA_LOG_RETENTION_HOURS: 24
      KAFKA_LOG_SEGMENT_BYTES: 1073741824
      KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 300000
      # ===== Network Threads =====
      KAFKA_NUM_NETWORK_THREADS: 4
      KAFKA_NUM_IO_THREADS: 8
      KAFKA_SOCKET_SEND_BUFFER_BYTES: 102400
      KAFKA_SOCKET_RECEIVE_BUFFER_BYTES: 102400
      KAFKA_SOCKET_REQUEST_MAX_BYTES: 104857600
      # ===== JMX =====
      JMX_PORT: 9999
      KAFKA_JMX_OPTS: >-
        -Dcom.sun.management.jmxremote=true
        -Dcom.sun.management.jmxremote.authenticate=false
        -Dcom.sun.management.jmxremote.ssl=false
        -Dcom.sun.management.jmxremote.rmi.port=9999
        -Djava.rmi.server.hostname=10.11.18.13
    volumes:
      - /data/kafka/data:/opt/kafka/kraft-combined-logs
    ulimits:
      nofile:
        soft: 65535
        hard: 65535

Node 2 (10.11.18.14) and Node 3 (10.11.18.15)

Configuration is identical except for the following per-node values:

container_name : kafka-node-2 / kafka-node-3

KAFKA_NODE_ID : 2 / 3

KAFKA_ADVERTISED_LISTENERS : PLAINTEXT://10.11.18.14:9092 / PLAINTEXT://10.11.18.15:9092

java.rmi.server.hostname (in KAFKA_JMX_OPTS): 10.11.18.14 / 10.11.18.15

All other environment variables, including CLUSTER_ID and KAFKA_CONTROLLER_QUORUM_VOTERS, must remain exactly the same across all three nodes.

Start the Cluster

On each machine, run:

cd /data/kafka
docker compose up -d

Verify the Cluster

Verification can be performed from any node.

1. Check Broker List

docker exec -it -e JMX_PORT="" kafka-node-1 /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092

Seeing nodes 1, 2, and 3 indicates a healthy cluster.

2. Create a Test Topic

Create a topic with 3 partitions and replication factor 3:

docker exec -it -e JMX_PORT="" kafka-node-1 /opt/kafka/bin/kafka-topics.sh --create --topic test-topic --partitions 3 --replication-factor 3 --bootstrap-server localhost:9092

3. Inspect Topic Details

docker exec -it -e JMX_PORT="" kafka-node-1 /opt/kafka/bin/kafka-topics.sh --describe --topic test-topic --bootstrap-server localhost:9092

Expected output shows ISR (In-Sync Replicas) as 1,2,3, confirming all three replicas are synchronized.

4. Produce Test Messages

docker exec -it -e JMX_PORT="" kafka-node-1 /opt/kafka/bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092

Type hello kafka cluster, then press Ctrl+C to exit.

5. Consume Test Messages

In a separate terminal:

docker exec -it -e JMX_PORT="" kafka-node-1 /opt/kafka/bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092

Receiving the previously sent message confirms end-to-end functionality.