How to Build a Production‑Ready ELK Logging Stack on Kubernetes

Main Content

ELK concepts

Which logs Kubernetes needs to collect

ELK Stack logging solution

How to collect logs inside containers

Deployment steps

Environment Preparation

A functional Kubernetes cluster (installed via kubeadm or binary) is required.

1. ELK Concepts

ELK stands for Elasticsearch, Logstash, and Kibana. Elasticsearch is a distributed, near‑real‑time search engine built on Lucene. Logstash is the central data‑flow engine that collects, filters, and forwards data. Kibana visualizes Elasticsearch data. ELK is often called the Elastic Stack and can be used for log analysis as well as other data‑collection scenarios.

Official site: https://www.elastic.co/cn/products/

2. Log Management Platform

In monolithic applications, logs were viewed via SSH on a single server. With micro‑services and large user bases, logs are distributed across many nodes, making a centralized log platform essential. The typical pipeline is: Logstash collects logs from each node, filters them, forwards to Kafka or Redis, another Logstash instance reads from Kafka/Redis and stores them in Elasticsearch, and Kibana visualizes the data for developers and operators.

3. ELK Stack Logging Scheme in Kubernetes

Three collection approaches are presented:

Option 1: Deploy a logging agent on each node (DaemonSet)

Deploy a logging program (logging‑agent) on every node to collect logs from /var/log and /var/lib/docker/containers (or mount pod log directories to a host directory).

Option 2: Add a sidecar container to each pod

Attach a dedicated log‑collector container to each application pod, sharing the log directory via an emptyDir volume.

Option 3: Application pushes logs directly

Modify application code to push logs to a remote store, which is less common in Kubernetes.

Pros and Cons

Option 1 : Low resource consumption, no intrusion; drawback – requires applications to write to stdout/stderr and does not support multi‑line logs.

Option 2 : Low coupling; drawback – extra sidecar per pod increases resource usage and operational cost.

Option 3 : No extra collector needed; drawback – invasive, increases application complexity.

4. Log Collection Issues in Kubernetes

Question 1: Which logs need to be collected?

Typical logs include Kubernetes system component logs and application logs (both stdout and log files).

Question 2: Where are the logs and how to collect them for common runtimes?

For Docker, logs are stored under /var/lib/docker/containers/$CONTAINERID with symlinks in /var/log/pods and /var/log/containers. For containerd, logs are stored under /var/log/pods/$CONTAINER_NAME with similar symlinks. Configuration parameters differ (Docker uses log-driver: json-file and size options; containerd is configured via kubelet flags).

Question 3: Should logs be standardized?

Recommended JSON format, one line per log. Required fields: level (debug, info, warn, error, fatal), msg, remote_ip, project, time (UTC), func. Optional fields include request_url, status, cost, method, _id. Ingress logs should follow a JSON template such as:

log-format-upstream: '{