How to Deploy a Scalable Kafka Cluster on Bare Metal and Kubernetes

Overview

Kafka requires at least three brokers for high availability, typically paired with ZooKeeper unless using the KRaft mode introduced in Kafka 3.3+. The guide presents two deployment approaches—direct installation on Linux servers (bare‑metal or virtual machines) and a cloud‑native solution using Kubernetes with Helm charts.

Bare‑Metal / Virtual Machine Deployment

1. Environment Preparation

Server count: Minimum three machines (physical or VM).

Java: JDK 8+ (Kafka 3.x also supports JDK 17).

Download Kafka:

wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
 tar -xzf kafka_2.13-3.7.0.tgz -C /opt/
 cd /opt/kafka_2.13-3.7.0

2. ZooKeeper Configuration

Edit /opt/kafka_2.13-3.7.0/config/zookeeper.properties with the following settings:

tickTime=2000
initLimit=5
syncLimit=2
dataDir=/var/lib/zookeeper
clientPort=2181
server.1=192.168.1.101:2888:3888
server.2=192.168.1.102:2888:3888
server.3=192.168.1.103:2888:3888

On each node, write the node’s ID (1, 2, 3) to /var/lib/zookeeper/myid and start ZooKeeper:

bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

3. Kafka Broker Configuration

On every machine, edit config/server.properties:

broker.id=1               # unique per node
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://192.168.1.101:9092
log.dirs=/var/lib/kafka-logs
zookeeper.connect=192.168.1.101:2181,192.168.1.102:2181,192.168.1.103:2181

Start each broker:

bin/kafka-server-start.sh -daemon config/server.properties

Kubernetes + Helm Deployment

This method is suited for cloud‑native environments and large‑scale production.

1. Add Bitnami Helm Repository

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

2. Create a Custom values.yaml

Example configuration that uses HostPath storage, exposes services via NodePort, and pulls images from a domestic mirror:

zookeeper:
  enabled: true
  replicaCount: 3
  persistence:
    enabled: true
    size: 5Gi
    hostPath: /data/zookeeper
  image:
    registry: docker.m.daocloud.io
    repository: bitnami/zookeeper
    tag: 3.9.1
kafka:
  replicaCount: 3
  listeners:
    client:
      protocol: PLAINTEXT
      port: 9092
      nodePort: 30092
  externalAccess:
    enabled: true
    service:
      type: NodePort
    nodePorts:
      - 30092
      - 30093
      - 30094
  persistence:
    enabled: true
    size: 10Gi
    hostPath: /data/kafka
  image:
    registry: docker.m.daocloud.io
    repository: bitnami/kafka
    tag: 3.7.0

3. Install Kafka via Helm

helm install kafka-cluster bitnami/kafka -f values.yaml

4. Verify Deployment

kubectl get pods -l app.kubernetes.io/name=kafka
kubectl get svc kafka-cluster

KRaft Mode (ZooKeeper‑Free)

For Kafka 3.3+ you can enable the built‑in KRaft mode, eliminating ZooKeeper and reducing operational overhead. Set process.roles=broker,controller in the broker configuration and adjust the cluster‑wide settings accordingly.

Production‑Grade Recommendations

Partitions & Replication: Use three replicas per topic.

Broker Memory: Allocate at least 4 GB heap per broker.

Disk: Prefer SSDs because Kafka is I/O‑intensive.

Monitoring: Deploy Prometheus + Grafana for metrics collection.

Automatic Rebalancing: Consider Cruise Control for load‑aware partition rebalancing.

Illustrative Diagram