How to Efficiently Collect Container Logs in Kubernetes: Strategies and Pitfalls

1. From Host to Container

In traditional VM or physical host environments, logs are written directly to the host and agents are manually configured per node. In Kubernetes, pods can migrate, logs are stored in various ways (stdout, hostPath, emptyDir, PV), and agents must inject metadata like namespace, pod, container, node, labels, and environment variables.

2.1 Types of Logs

Cloud‑native best practice recommends outputting logs to stdout, but many applications still write to files for audit or access logs, making collection more complex.

2.2 Agent Deployment Methods

Two main deployment models exist:

DaemonSet : one agent per node.

Sidecar : an agent container added to each pod.

Key trade‑offs:

Resource usage : DaemonSet scales with nodes, Sidecar scales with pods, leading to higher overhead on dense nodes.

Intrusiveness : Sidecar injects an extra container into business pods, potentially altering deployment.

Stability : Sidecar failures can affect the business container; many agents may also stress downstream services like Kafka.

Isolation : Sidecar isolates logs per pod, while DaemonSet shares a single agent per node.

Performance : Sidecar handles less log volume per agent, reducing the chance of hitting performance limits.

Tip: Prefer DaemonSet for most cases; use Sidecar only when a single pod generates exceptionally high log volume.

2.3 Collection Approaches

DaemonSet + Stdout

When Docker is used, container stdout logs are stored under

/var/lib/docker/containers/<containerId>/<containerId>-json.log

. Prior to Kubernetes 1.14, kubelet created symlinks in /var/log/pods pointing to these files. From 1.14 onward, the path changed to

/var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/<num>.log

.

root@master0:/var/log/pods# tree .
|-- 6687e53201c01e3fad31e7d72fbb92a6
|   `-- kube-apiserver
|       |-- 865.log -> /var/lib/docker/containers/3a35ae0a1d0b26455fbd9b267cd9d6ac3fbd3f0b12ee03b4b22b80dc5a1cde03/3a35ae0a1d0b26455fbd9b267cd9d6ac3fbd3f0b12ee03b4b22b80dc5a1cde03-json.log
|       `-- 866.log -> /var/lib/docker/containers/15a6924f14fcbf15dd37d1c185c5b95154fa2c5f3de9513204b1066bbe474662/15a6924f14fcbf15dd37d1c185c5b95154fa2c5f3de9513204b1066bbe474662-json.log
|-- a1083c6d-3b12-11ea-9af1-fa163e28f309
|   `-- kube-proxy
|       |-- 3.log -> /var/lib/docker/containers/4b63b5a90a8f9ca6b6f20b49b5ab2564f92df21a5590f46de2a46b031e55c80e/4b63b5a90a8f9ca6b6f20b49b5ab2564f92df21a5590f46de2a46b031e55c80e-json.log
|       `-- 4.log -> /var/lib/docker/containers/fc7c315d33935887ca3479a38cfca4cca66fad782b8a120c548ad0b9f0ff7207/fc7c315d33935887ca3479a38cfca4cca66fad782b8a120c548ad0b9f0ff7207-json.log

After 1.14, the layout is:

root@master-0:/var/log/pods# tree .
|-- kube-system_kube-apiserver-kind-control-plane_bd1c21fe1f0ef615e0b5e41299f1be61
|   `-- kube-apiserver
|       `-- 0.log
|-- kube-system_kube-proxy-gcrfq_f07260b8-6055-4c19-9491-4a825579528f
|   `-- kube-proxy
|       `-- 0.log
|-- loggie_loggie-csd4g_f1cc32e9-1002-4e64-bd58-fc6094394e06
|    `-- loggie
|        `-- 0.log

Deploy a DaemonSet that mounts /var/log/pods and configures the agent to watch /var/log/pod.log (or similar glob patterns) to collect all container stdout logs.

Tip: This simple approach works for basic scenarios but cannot inject additional metadata or handle per‑service customizations.

DaemonSet + Log Files

If applications write logs to files, the agent must access those files on the node. Common volume types:

emptyDir : lifecycle tied to the pod; logs disappear when the pod is deleted. Path example:

/var/lib/kubelet/pods/${pod.UID}/volumes/kubernetes.io~empty-dir/${volumeName}

.

hostPath : persists on the node independent of pod lifecycle; requires careful path management and may cause disk bloat.

PersistentVolume (PV) : supports ReadWriteOnce, ReadOnlyMany, ReadWriteMany; suitable for stateful workloads but adds operational complexity and potential isolation issues.

Agents must mount the same host paths as the application pods to read the log files. Using subPathExpr (enabled by default from Kubernetes 1.15) helps isolate logs per pod when using hostPath.

Overall, simple DaemonSet‑based collection works for straightforward use cases, but more advanced requirements (metadata injection, selective processing, high‑volume pods) often necessitate sidecar agents or specialized log collectors like Filebeat or Fluentd.