How to Build a Production‑Ready ELK Logging Stack on Kubernetes

Main Content

ELK concepts

Which logs a K8s cluster needs

ELK Stack logging solution

How to collect logs inside containers

Deployment steps

Environment Preparation

A functional Kubernetes cluster (installed via kubeadm or binary) is required.

1. ELK Concepts

ELK stands for Elasticsearch, Logstash, and Kibana, collectively known as the Elastic Stack. Elasticsearch is a distributed, near‑real‑time search engine built on Lucene. Logstash is the central data‑flow engine that collects, filters, and forwards data. Kibana visualizes Elasticsearch data through a web UI.

2. Log Management Platform

In monolithic applications, logs were inspected directly on a single server. Modern micro‑service architectures deployed on many nodes require a centralized log platform. Logstash collects logs from each node, filters them, forwards to Kafka or Redis, stores them in Elasticsearch, and visualizes them with Kibana, greatly improving incident response.

3. ELK Stack Logging in Kubernetes

Three collection approaches are presented:

Option 1 – Deploy a logging agent on each node (DaemonSet) : A logging‑agent runs on every node, reading /var/log and /var/lib/docker/containers (or the equivalent for containerd).

Option 2 – Add a sidecar container to each pod : A dedicated logging container shares the pod’s log directory via an emptyDir volume.

Option 3 – Application pushes logs directly : Applications write logs to a remote store, bypassing local files (outside the Kubernetes scope).

Pros and cons of each method:

Option 1: Low resource usage, no app intrusion; downside – requires logs on stdout/stderr and cannot handle multi‑line logs.

Option 2: Low coupling; downside – extra pod per application, higher resource consumption.

Option 3: No extra collector needed; downside – adds complexity to the application.

4. Log Collection Considerations in K8s

Question 1: Which logs to collect?

Collect both Kubernetes component logs and application logs (stdout and log files).

Question 2: Where are the logs and how to collect them for Docker vs. containerd?

For Docker, logs reside in /var/lib/docker/containers/$CONTAINERID with symlinks under /var/log/pods and /var/log/containers. For containerd, logs are stored directly under /var/log/pods/$CONTAINER_NAME with similar symlinks.

Question 3: Should logs be standardized?

Use a single‑line JSON format with required fields such as level (debug, info, warn, error, fatal), msg, remote_ip, project, time (UTC), and func. Optional fields include request_url, status, cost, method, and _id. Ingress logs should follow a predefined JSON schema.

5. Deployment Steps

5.1 Single‑Node Elasticsearch Deployment

Use a StatefulSet YAML to create a single Elasticsearch node, expose it via a Service, and label the node for persistent storage.

# Label the node for ES storage
kubectl label node xxxx es=data
# es.yaml (truncated for brevity)
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-logging
  namespace: kube-system
spec:
  ports:
  - port: 9200
    protocol: TCP
    targetPort: db
  selector:
    k8s-app: elasticsearch-logging
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch-logging
  namespace: kube-system
spec:
  serviceName: elasticsearch-logging
  replicas: 1
  selector:
    matchLabels:
      k8s-app: elasticsearch-logging
  template:
    metadata:
      labels:
        k8s-app: elasticsearch-logging
    spec:
      containers:
      - name: elasticsearch-logging
        image: docker.io/library/elasticsearch:7.9.3
        env:
        - name: discovery.type
          value: "single-node"
        ports:
        - containerPort: 9200
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      volumes:
      - name: data
        hostPath:
          path: /data/es/

5.2 Deploy Kibana

Create a Service (NodePort 25601) and a Deployment that points to the Elasticsearch service.

# kibana.yaml (truncated)
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - port: 5601
    nodePort: 25601
    targetPort: ui
  selector:
    k8s-app: kibana
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana
  template:
    metadata:
      labels:
        k8s-app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.io/kubeimages/kibana:7.9.3
        env:
        - name: ELASTICSEARCH_HOSTS
          value: http://elasticsearch-logging:9200
        ports:
        - containerPort: 5601

5.3 Deploy Filebeat on Each Node

Use a DaemonSet to run Filebeat, mounting host log directories and forwarding logs to Logstash.

# filebeat.yaml (truncated)
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
data:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths:
      - /var/log/containers/*.log
      processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
    output.logstash:
      hosts: ["logstash:5044"]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      containers:
      - name: filebeat
        image: docker.io/kubeimages/filebeat:7.9.3
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: data
          mountPath: /usr/share/filebeat/data
      volumes:
      - name: varlog
        hostPath:
          path: /var/log

5.4 Deploy Logstash for Log Enrichment

Logstash parses JSON logs, extracts fields, converts types, and writes enriched logs to Elasticsearch.

# logstash.yaml (truncated)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash
  namespace: kube-system
spec:
  selector:
    matchLabels:
      type: logstash
  template:
    metadata:
      labels:
        type: logstash
    spec:
      containers:
      - name: logstash
        image: docker.io/kubeimages/logstash:7.9.3
        ports:
        - containerPort: 5044
        command: ["logstash", "-f", "/etc/logstash_c/logstash.conf"]
        env:
        - name: XPACK_MONITORING_ELASTICSEARCH_HOSTS
          value: "http://elasticsearch-logging:9200"
        volumeMounts:
        - name: config-volume
          mountPath: /etc/logstash_c/
      volumes:
      - name: config-volume
        configMap:
          name: logstash-conf
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-conf
  namespace: kube-system
data:
  logstash.conf: |
    input { beats { port => 5044 } }
    filter {
      if [kubernetes][container][name] == "nginx-ingress-controller" {
        json { source => "message" target => "ingress_log" }
        # type conversions omitted for brevity
      }
      if [kubernetes][container][name] =~ /^srv/ {
        json { source => "message" target => "tmp" }
        # field extraction and renaming omitted for brevity
      }
      mutate { rename => { "kubernetes" => "k8s" } }
    }
    output { elasticsearch { hosts => ["http://elasticsearch-logging:9200"] index => "logstash-%{+YYYY.MM.dd}" } }

5.5 Kibana UI Configuration

After accessing http://<node_ip>:25601, create an index pattern matching logstash-*, set the time field to @timestamp, and optionally define an Index Lifecycle Policy (e.g., logstash-history-ilm-policy) to manage retention.

5.6 Adjust Index Settings for a Single‑Node Cluster

Set number_of_replicas to 0 for all indices and define a custom template that includes the lifecycle policy and proper mappings for the enriched applog and k8s fields.

# Reduce replicas
PUT _all/_settings {
  "number_of_replicas": 0
}
# Define template with lifecycle policy
PUT _template/logstash {
  "order": 1,
  "index_patterns": ["logstash-*"] ,
  "settings": {
    "index.lifecycle.name": "logstash-history-ilm-policy",
    "number_of_shards": "2",
    "number_of_replicas": "0"
  },
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "applog": { "dynamic": true, "properties": { "cost": { "type": "float" }, "func": { "type": "keyword" }, "method": { "type": "keyword" } } },
      "k8s": { "dynamic": true, "properties": { "namespace": { "type": "keyword" }, "container": { "properties": { "name": { "type": "keyword" } } } } }
    }
  }
}

With these components in place, the Kubernetes cluster now has a fully functional, searchable, and visualizable logging pipeline suitable for production environments.