How to Build a Production-Ready ELK Logging Platform on Kubernetes

Main Content

ELK concepts

Which logs Kubernetes needs to collect

ELK Stack logging solutions

How to collect logs inside containers

Kubernetes application log collection

Preparation Environment

A running Kubernetes cluster (installed via kubeadm or binary) is required.

Example node IPs and roles:

192.168.73.136 – nfs 192.168.73.138 – k8s-master 192.168.73.139 – k8s-node01 192.168.73.140 – k8s-node02

1. ELK Concepts

ELK stands for Elasticsearch, Logstash, and Kibana, also known as the Elastic Stack. Elasticsearch is a distributed, real‑time search engine built on Lucene. Logstash is the central data‑flow engine that collects, filters, and forwards data to various destinations. Kibana visualizes Elasticsearch data through a web UI. While ELK is often used for log analysis, it can handle any data collection and analysis scenario.

Official site: https://www.elastic.co/cn/products/

2. Log Management Platform

In monolithic applications, logs were accessed directly on a single server. With microservices and horizontal scaling, logs are distributed across many nodes, making a centralized log management platform essential. The typical pipeline uses Logstash to collect logs from each server, filter them, forward to Kafka or Redis, then another Logstash instance writes them to Elasticsearch, and Kibana visualizes the data, greatly improving incident response and enabling big‑data analysis for business decisions.

3. Which Logs Kubernetes Needs to Collect

Mainly:

Kubernetes system component logs

Application logs running in the cluster (standard output and log files)

4. ELK Stack Log Collection Schemes in Kubernetes

Scheme 1: Deploy a logging agent on each node using a DaemonSet (logging‑agent) to collect logs from /var/log and /var/lib/docker/containers or mount pod log directories to the host.

Scheme 2: Add a dedicated log‑collector container to each pod, sharing the log directory via emptyDir so the collector can read the logs.

Scheme 3: Modify the application code to push logs directly to a remote store, bypassing the container’s stdout/stderr. This approach is less common and goes beyond typical Kubernetes logging.

5. Single‑Node ELK Deployment

Deploy a single‑node Elasticsearch and Kibana using the following YAML files. For log volumes exceeding ~20 GB per day, a distributed deployment outside the cluster is recommended.

[root@k8s-master fek]# vim elasticsearch.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: kube-system
  labels:
    k8s-app: elasticsearch
spec:
  serviceName: elasticsearch
  selector:
    matchLabels:
      k8s-app: elasticsearch
  template:
    metadata:
      labels:
        k8s-app: elasticsearch
    spec:
      containers:
      - image: elasticsearch:7.3.1
        name: elasticsearch
        resources:
          limits:
            cpu: 1
            memory: 2Gi
          requests:
            cpu: 0.5
            memory: 500Mi
        env:
          - name: "discovery.type"
            value: "single-node"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx2g"
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
  - metadata:
      name: elasticsearch-data
    spec:
      storageClassName: "managed-nfs-storage"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 20Gi

---

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: kube-system
spec:
  clusterIP: None
  ports:
  - port: 9200
    protocol: TCP
    targetPort: db
  selector:
    k8s-app: elasticsearch

Apply the file and verify the Elasticsearch pod is running.

[root@k8s-master fek]# kubectl get pod -n kube-system
NAME                        READY   STATUS    RESTARTS   AGE
elasticsearch-0             1/1     Running   1          16m

Next, deploy Kibana:

[root@k8s-master fek]# vim kibana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: kube-system
  labels:
    k8s-app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana
  template:
    metadata:
      labels:
        k8s-app: kibana
    spec:
      containers:
      - name: kibana
        image: kibana:7.3.1
        resources:
          limits:
            cpu: 1
            memory: 500Mi
          requests:
            cpu: 0.5
            memory: 200Mi
        env:
          - name: ELASTICSEARCH_HOSTS
            value: http://elasticsearch:9200
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: kube-system
spec:
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana
  namespace: kube-system
spec:
  rules:
  - host: kibana.ctnrs.com
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana
          servicePort: 5601

[root@k8s-master fek]# kubectl apply -f kibana.yaml
deployment.apps/kibana created
service/kibana created
ingress.extensions/kibana created

[root@k8s-master fek]# kubectl get pod -n kube-system
NAME                        READY   STATUS    RESTARTS   AGE
kibana-b7d98644-48gtm       1/1     Running   1          17h

5.1 Scheme 1: Deploy Filebeat on Nodes to Collect Kubernetes Component Logs

After Elasticsearch and Kibana are up, deploy a Filebeat DaemonSet on each node (version 7.3.1) to collect pod logs and component logs from /var/log/messages.

[root@k8s-master fek]# vim filebeat-kubernetes.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat

data:
  filebeat.yml: |-
    filebeat.config:
      inputs:
        path: ${path.config}/inputs.d/*.yml
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-inputs
  namespace: kube-system
  labels:
    k8s-app: filebeat

data:
  kubernetes.yml: |-
    - type: docker
      containers.ids:
      - "*"
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: elastic/filebeat:7.3.1
        args: ["-c", "/etc/filebeat.yml", "-e"]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          subPath: filebeat.yml
        - name: inputs
          mountPath: /usr/share/filebeat/inputs.d
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: inputs
        configMap:
          defaultMode: 0600
          name: filebeat-inputs
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate

Apply the manifest and verify the Filebeat pods are running.

[root@k8s-master elk]# kubectl apply -f filebeat-kubernetes.yaml
... (output omitted) ...
[root@k8s-master elk]# kubectl get pod -n kube-system
NAME                        READY   STATUS    RESTARTS   AGE
filebeat-2q5tz              1/1     Running   0          13h
filebeat-k6m27              1/1     Running   2          13h

5.1.1 Visualizing Logs in Kibana

Open the Kibana web UI, navigate to *Management → Index Patterns*, and create index patterns for filebeat-7.3.1-* and k8s-module-*. After setting the time filter, go to *Discover* to view collected logs.

5.2 Scheme 2: Add a Dedicated Log‑Collector Container to Pods

Inject a Filebeat sidecar into application pods (example: a PHP demo) using an emptyDir volume to share the log directory.

[root@k8s-master fek]# vim nginx-deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: php-demo
  namespace: kube-system
spec:
  replicas: 2
  selector:
    matchLabels:
      project: www
      app: php-demo
  template:
    metadata:
      labels:
        project: www
        app: php-demo
    spec:
      containers:
      - name: nginx
        image: lizhenliang/nginx-php
        ports:
        - containerPort: 80
          name: web
          protocol: TCP
        volumeMounts:
        - name: nginx-logs
          mountPath: /usr/local/nginx/logs
      - name: filebeat
        image: elastic/filebeat:7.3.1
        args: ["-c", "/etc/filebeat.yml", "-e"]
        securityContext:
          runAsUser: 0
        volumeMounts:
        - name: filebeat-config
          mountPath: /etc/filebeat.yml
          subPath: filebeat.yml
        - name: nginx-logs
          mountPath: /usr/local/nginx/logs
      volumes:
      - name: nginx-logs
        emptyDir: {}
      - name: filebeat-config
        configMap:
          name: filebeat-nginx-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-nginx-config
  namespace: kube-system

data:
  filebeat.yml: |-
    filebeat.inputs:
      - type: log
        paths:
          - /usr/local/nginx/logs/access.log
        fields:
          app: www
          type: nginx-access
        fields_under_root: true
    setup.ilm.enabled: false
    setup.template.name: "nginx-access"
    setup.template.pattern: "nginx-access-*"
    output.elasticsearch:
      hosts: ['elasticsearch.kube-system:9200']
      index: "nginx-access-%{+yyyy.MM.dd}"

Deploy the manifest and verify the pods and sidecar containers are running.

[root@k8s-master elk]# kubectl apply -f nginx-deployment.yaml
... (output omitted) ...
[root@k8s-master elk]# kubectl get pod -n kube-system
NAME                        READY   STATUS    RESTARTS   AGE
php-demo-85849d58df-d98gv   2/2     Running   0          26m
php-demo-85849d58df-sl5ss   2/2     Running   0          26m

In Kibana, create an index pattern for nginx-access-* and explore the logs.

Conclusion

The guide demonstrates how to set up a production‑grade ELK logging stack on a Kubernetes cluster, covering component installation, three collection strategies, Filebeat deployment, ingress exposure, and Kibana visualization. This provides a solid foundation for DevOps teams to monitor, troubleshoot, and analyze logs at scale.