Deploy Zookeeper and Kafka on Kubernetes with Helm: A Complete Step‑by‑Step Guide

1. Overview

Apache ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization and group services; it can be viewed as a distributed database with a tree‑like structure. Official documentation: https://zookeeper.apache.org/doc/r3.8.0/

Kafka was originally developed by LinkedIn; it is a distributed, partitioned, multi‑replica messaging system that relies on ZooKeeper for coordination. Official documentation: https://kafka.apache.org/documentation/

2. Deploy Zookeeper on Kubernetes

1) Add repository

Deployment package address:

https://artifacthub.io/packages/helm/zookeeper/zookeeper

helm repo add bitnami https://charts.bitnami.com/bitnami
helm pull bitnami/zookeeper
tar -xf  zookeeper-10.2.1.tgz

2) Modify configuration

Edit zookeeper/values.yaml :

image:
  registry: myharbor.com
  repository: bigdata/zookeeper
  tag: 3.8.0-debian-11-r36
...
replicaCount: 3
...
service:
  type: NodePort
  nodePorts:
    # NodePort default range is 30000-32767
    client: "32181"
    tls: "32182"
...

persistence:
  storageClass: "zookeeper-local-storage"
  size: "10Gi"
  # Directory must be created on the host beforehand
  local:
    - name: zookeeper-0
      host: "local-168-182-110"
      path: "/opt/bigdata/servers/zookeeper/data/data1"
    - name: zookeeper-1
      host: "local-168-182-111"
      path: "/opt/bigdata/servers/zookeeper/data/data1"
    - name: zookeeper-2
      host: "local-168-182-112"
      path: "/opt/bigdata/servers/zookeeper/data/data1"
...
# Enable Prometheus to access ZooKeeper metrics endpoint
metrics:
  enabled: true

Add zookeeper/templates/pv.yaml :

{{- range .Values.persistence.local }}
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: {{ .name }}
  labels:
    name: {{ .name }}
spec:
  storageClassName: {{ $.Values.persistence.storageClass }}
  capacity:
    storage: {{ $.Values.persistence.size }}
  accessModes:
    - ReadWriteOnce
  local:
    path: {{ .path }}
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - {{ .host }}
---
{{- end }}

Add zookeeper/templates/storage-class.yaml :

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: {{ .Values.persistence.storageClass }}
provisioner: kubernetes.io/no-provisioner

3) Install

# Prepare images
docker pull docker.io/bitnami/zookeeper:3.8.0-debian-11-r36
docker tag docker.io/bitnami/zookeeper:3.8.0-debian-11-r36 myharbor.com/bigdata/zookeeper:3.8.0-debian-11-r36
docker push myharbor.com/bigdata/zookeeper:3.8.0-debian-11-r36

# Install chart
helm install zookeeper ./zookeeper -n zookeeper --create-namespace

Notes:

NAME: zookeeper
LAST DEPLOYED: Sun Sep 18 18:24:03 2022
NAMESPACE: zookeeper
STATUS: deployed
REVISION: 1
CHART NAME: zookeeper
CHART VERSION: 10.2.1
APP VERSION: 3.8.0

** Please be patient while the chart is being deployed **

ZooKeeper can be accessed via port 2181 on the DNS name zookeeper.zookeeper.svc.cluster.local inside the cluster.

# Connect from inside the cluster
export POD_NAME=$(kubectl get pods -n zookeeper -l "app.kubernetes.io/name=zookeeper,app.kubernetes.io/instance=zookeeper,app.kubernetes.io/component=zookeeper" -o jsonpath="{.items[0].metadata.name}")
kubectl exec -it $POD_NAME -- zkCli.sh

# Connect from outside the cluster
export NODE_IP=$(kubectl get nodes -n zookeeper -o jsonpath="{.items[0].status.addresses[0].address}")
export NODE_PORT=$(kubectl get -n zookeeper -o jsonpath="{.spec.ports[0].nodePort}" services zookeeper)
zkCli.sh $NODE_IP:$NODE_PORT

kubectl get pods,svc -n zookeeper -owide

4) Test verification

# Login to Zookeeper pod
kubectl exec -it zookeeper-0 -n zookeeper -- zkServer.sh status
kubectl exec -it zookeeper-1 -n zookeeper -- zkServer.sh status
kubectl exec -it zookeeper-2 -n zookeeper -- zkServer.sh status

# Open a shell in a pod
kubectl exec -it zookeeper-0 -n zookeeper -- bash

5) Prometheus monitoring

Prometheus URL: https://prometheus.k8s.local/targets?search=zookeeper

Query metrics:

kubectl get --raw http://10.244.0.52:9141/metrics
kubectl get --raw http://10.244.1.101:9141/metrics
kubectl get --raw http://10.244.2.137:9141/metrics

Grafana URL: https://grafana.k8s.local/ (admin password obtained via

kubectl get secret -n grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode

)

6) Uninstall

helm uninstall zookeeper -n zookeeper

kubectl delete pod -n zookeeper `kubectl get pod -n zookeeper | awk 'NR>1{print $1}'` --force
kubectl patch ns zookeeper -p '{"metadata":{"finalizers":null}}'
kubectl delete ns zookeeper --force

3. Deploy Kafka on Kubernetes

1) Add repository

Deployment package address:

https://artifacthub.io/packages/helm/bitnami/kafka

helm repo add bitnami https://charts.bitnami.com/bitnami
helm pull bitnami/kafka
tar -xf kafka-18.4.2.tgz

2) Modify configuration

Edit kafka/values.yaml :

image:
  registry: myharbor.com
  repository: bigdata/kafka
  tag: 3.2.1-debian-11-r16
...
replicaCount: 3
...
service:
  type: NodePort
  nodePorts:
    client: "30092"
    external: "30094"
...
externalAccess:
  enabled: true
  service:
    type: NodePort
    nodePorts:
      - 30001
      - 30002
      - 30003
    useHostIPs: true
...

persistence:
  storageClass: "kafka-local-storage"
  size: "10Gi"
  # Directory must be created on the host beforehand
  local:
    - name: kafka-0
      host: "local-168-182-110"
      path: "/opt/bigdata/servers/kafka/data/data1"
    - name: kafka-1
      host: "local-168-182-111"
      path: "/opt/bigdata/servers/kafka/data/data1"
    - name: kafka-2
      host: "local-168-182-112"
      path: "/opt/bigdata/servers/kafka/data/data1"
...
metrics:
  kafka:
    enabled: true
    image:
      registry: myharbor.com
      repository: bigdata/kafka-exporter
      tag: 1.6.0-debian-11-r8
    jmx:
      enabled: true
      image:
        registry: myharbor.com
        repository: bigdata/jmx-exporter
        tag: 0.17.1-debian-11-r1
      annotations:
        prometheus.io/path: "/metrics"
...
zookeeper:
  enabled: false
...
externalZookeeper:
  servers:
    - zookeeper-0.zookeeper-headless.zookeeper
    - zookeeper-1.zookeeper-headless.zookeeper
    - zookeeper-2.zookeeper-headless.zookeeper

Add kafka/templates/pv.yaml (similar to Zookeeper PV template).

{{- range .Values.persistence.local }}
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: {{ .name }}
  labels:
    name: {{ .name }}
spec:
  storageClassName: {{ $.Values.persistence.storageClass }}
  capacity:
    storage: {{ $.Values.persistence.size }}
  accessModes:
    - ReadWriteOnce
  local:
    path: {{ .path }}
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - {{ .host }}
---
{{- end }}

Add kafka/templates/storage-class.yaml :

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: {{ .Values.persistence.storageClass }}
provisioner: kubernetes.io/no-provisioner

3) Install

# Prepare images
docker pull docker.io/bitnami/kafka:3.2.1-debian-11-r16
docker tag docker.io/bitnami/kafka:3.2.1-debian-11-r16 myharbor.com/bigdata/kafka:3.2.1-debian-11-r16
docker push myharbor.com/bigdata/kafka:3.2.1-debian-11-r16

# Exporter images
docker pull docker.io/bitnami/kafka-exporter:1.6.0-debian-11-r8
docker tag docker.io/bitnami/kafka-exporter:1.6.0-debian-11-r8 myharbor.com/bigdata/kafka-exporter:1.6.0-debian-11-r8
docker push myharbor.com/bigdata/kafka-exporter:1.6.0-debian-11-r8

docker pull docker.io/bitnami/jmx-exporter:0.17.1-debian-11-r1
docker tag docker.io/bitnami/jmx-exporter:0.17.1-debian-11-r1 myharbor.com/bigdata/jmx-exporter:0.17.1-debian-11-r1
docker push myharbor.com/bigdata/jmx-exporter:0.17.1-debian-11-r1

# Install chart
helm install kafka ./kafka -n kafka --create-namespace

Notes:

NAME: kafka
LAST DEPLOYED: Sun Sep 18 20:57:02 2022
NAMESPACE: kafka
STATUS: deployed
REVISION: 1
CHART NAME: kafka
CHART VERSION: 18.4.2
APP VERSION: 3.2.1

** Please be patient while the chart is being deployed **

Kafka can be accessed by consumers via port 9092 at kafka.kafka.svc.cluster.local inside the cluster.
Each broker is reachable at kafka-0.kafka-headless.kafka.svc.cluster.local:9092, etc.

# Create a client pod
kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.2.1-debian-11-r16 -n kafka --command -- sleep infinity
kubectl exec -it kafka-client -n kafka -- bash

# Produce messages
kafka-console-producer.sh --broker-list kafka-0.kafka-headless.kafka.svc.cluster.local:9092,kafka-1.kafka-headless.kafka.svc.cluster.local:9092,kafka-2.kafka-headless.kafka.svc.cluster.local:9092 --topic test

# Consume messages
kafka-console-consumer.sh --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic test --from-beginning

kubectl get pods,svc -n kafka -owide

4) Test verification

# Login to Kafka pod
kubectl exec -it kafka-0 -n kafka -- bash

1. Create Topic (one replica, one partition)

kafka-topics.sh --create --topic test001 --bootstrap-server kafka.kafka:9092 --partitions 1 --replication-factor 1
kafka-topics.sh --describe --bootstrap-server kafka.kafka:9092 --topic test001

2. List Topics

kafka-topics.sh --list --bootstrap-server kafka.kafka:9092

3. Producer / Consumer test

Producer:

kafka-console-producer.sh --broker-list kafka.kafka:9092 --topic test001
{"id":"1","name":"n1","age":"20"}
{"id":"2","name":"n2","age":"21"}
{"id":"3","name":"n3","age":"22"}

Consumer:

# Consume from beginning
kafka-console-consumer.sh --bootstrap-server kafka.kafka:9092 --topic test001 --from-beginning
# Consume from a specific offset
kafka-console-consumer.sh --bootstrap-server kafka.kafka:9092 --topic test001 --partition 0 --offset 100 --group test001

4. Check lag

kafka-consumer-groups.sh --bootstrap-server kafka.kafka:9092 --describe --group test001

5. Delete Topic

kafka-topics.sh --delete --topic test001 --bootstrap-server kafka.kafka:9092

5) Prometheus monitoring

Prometheus URL: https://prometheus.k8s.local/targets?search=kafka

Query metrics:

kubectl get --raw http://10.244.2.165:9308/metrics

Grafana URL: https://grafana.k8s.local/ (admin password via

kubectl get secret -n grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode

)

Import Grafana dashboard ID 11962 for cluster resource monitoring. Official dashboards: https://grafana.com/grafana/dashboards/

6) Uninstall

helm uninstall kafka -n kafka

kubectl delete pod -n kafka `kubectl get pod -n kafka | awk 'NR>1{print $1}'` --force
kubectl patch ns kafka -p '{"metadata":{"finalizers":null}}'
kubectl delete ns kafka --force

The Zookeeper + Kafka deployment on a Kubernetes environment is now complete; feel free to leave comments with any questions.