Deploy Filebeat with Helm on Kubernetes: Automated Log Collection to Kafka

In cloud‑native environments, log collection is essential for observability. This guide walks through deploying Filebeat via a Helm chart to automatically collect container logs from a Kubernetes cluster and forward them to a Kafka cluster.

1. Environment Preparation

Kubernetes cluster (v1.18+)

Helm package manager (v3.0+)

Kafka cluster (as log sink)

Private image registry (for custom images)

2. Chart Management

2.1 Get the official Chart

$ helm repo add elastic https://helm.elastic.co --force-update
"elastic" has been added to your repositories

$ helm pull elastic/filebeat --version 7.17.3

2.2 Push to private registry

$ helm push filebeat-7.17.3.tgz oci://core.jiaxzeng.com/plugins
Pushed: core.jiaxzeng.com/plugins/filebeat:7.17.3
Digest: sha256:76778389d4c793b414d392e9283851b7356feec9619dd37f0b7272c8ce42bf01

2.3 Load Chart locally

$ sudo helm pull oci://core.jiaxzeng.com/plugins/filebeat --version 7.17.3 --untar --untardir /etc/kubernetes/addons/
Pulled: core.jiaxzeng.com/plugins/filebeat:7.17.3
Digest: sha256:76778389d4c793b414d392e9283851b7356feec9619dd37f0b7272c8ce42bf01

3. Core Configuration

3.1 DaemonSet configuration

daemonset:
  enabled: true
  resources:
    requests:
      cpu: "100m"
      memory: "100Mi"
    limits:
      cpu: "1000m"
      memory: "200Mi"

DaemonSet must be used to ensure an instance runs on every node.

3.2 Log collection configuration

filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/*.log
  fields:
    type: k8s_logs
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
          - logs_path: "/var/log/containers/"
- type: log
  enabled: true
  fields:
    type: messages
  paths:
    - /var/log/messages

Path explanations: /var/log/containers/*.log – container stdout logs generated by Kubelet /var/log/messages – system logs (ensure the path exists on nodes)

3.3 Kafka output configuration

output.kafka:
  hosts: ["172.139.20.17:9092","172.139.20.81:9092","172.139.20.177:9092"]
  topics:
    - topic: 'k8s_logs'
      when.equals:
        fields.type: k8s_logs
    - topic: 'messages'
      when.equals:
        fields.type: messages
  partition.round_robin: {}
  reachable_only: true
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

3.4 Full values file (excerpt)

fullnameOverride: "filebeat"
image: "core.jiaxzeng.com/library/filebeat"
hostPathRoot: /var/lib
tolerations:
  - effect: NoSchedule
    operator: Exists
daemonset:
  enabled: true
  resources:
    requests:
      cpu: "100m"
      memory: "100Mi"
    limits:
      cpu: "1000m"
      memory: "200Mi"
filebeatConfig:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      fields:
        type: k8s_logs
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
              - logs_path: "/var/log/containers/"
    - type: log
      enabled: true
      fields:
        type: messages
      paths:
        - /var/log/messages
output.kafka:
  hosts: ["172.139.20.17:9092","172.139.20.81:9092","172.139.20.177:9092"]
  topics:
    - topic: 'k8s_logs'
      when.equals:
        fields.type: k8s_logs
    - topic: 'messages'
      when.equals:
        fields.type: messages
  partition.round_robin: {}
  reachable_only: true
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

4. Deployment

4.1 Install command

$ helm -n obs-system install filebeat -f /etc/kubernetes/addons/filebeat-values.yaml /etc/kubernetes/addons/filebeat

4.2 Verify DaemonSet pods

$ kubectl -n obs-system get pods -l app=filebeat -o wide
NAME          READY STATUS  RESTARTS AGE IP            NODE
filebeat-...  1/1   Running 1 (69m ago) 8h 10.244...   k8s-node04
...

5. Data Validation

5.1 Check Kafka consumer offsets

$ ./kafka-get-offsets.sh --bootstrap-server 172.139.20.17:9092 --topic k8s_logs
k8s_logs:0:296025
k8s_logs:1:297971
k8s_logs:2:297818
k8s_logs:3:296924
k8s_logs:4:296992
k8s_logs:5:297129

6. Conclusion

Mastering Helm‑based Filebeat deployment empowers efficient log handling in containerized environments, enabling reliable data pipelines to Kafka.