How to Build a Scalable Kubernetes Log Collection System with S6 and Filebeat

Container Logs Overview

Docker distinguishes between engine logs (handled by the host system) and container logs, which capture an application’s stdout and stderr. By default, container logs are stored as JSON files under

/var/lib/docker/containers/<container_id>/<container_id>-json.log

, a format unsuitable for production due to unlimited growth and daemon bottlenecks.

Problems with Default Logging

Log files grow without size limits, eventually filling the disk.

Docker daemon becomes a collection bottleneck when log volume is high.

Commands like docker logs -f can block the daemon, affecting other Docker commands.

Logging Drivers and Performance

Docker offers configurable logging drivers, but they still route logs through the daemon. Benchmarks show:

syslog: 14.9 MB/s

json‑file: 37.9 MB/s

Redirecting Logs with S6

Using an S6‑based base image, the container’s stdout/stderr can be redirected to files (e.g., /data/logs/.../app.log) with automatic rotation, bypassing the Docker daemon and eliminating its performance bottleneck.

Kubernetes Logging Levels

Kubernetes logging can be organized into three levels:

Application (Pod) level : Logs go to stdout/stderr and are accessed via kubectl logs.

Node level : Configure container log‑driver and use tools like logrotate for file rotation.

Cluster level : Deploy a node‑agent (DaemonSet) or sidecar containers to collect logs centrally.

Node‑Agent Architecture

A DaemonSet runs a log‑agent on each node. The agent includes Filebeat for file collection, Logrotate for rotation, and forwards logs to Kafka. Kafka feeds Elasticsearch via Logstash, enabling Kibana visualisation.

Implementation Steps

Build all application containers on the S6 base image so logs are written to host‑mounted files.

Deploy a log‑agent DaemonSet containing Filebeat, Logrotate, and custom components.

Use fsnotify to watch log directories and dynamically update Filebeat configuration.

Schedule periodic log rotation with cron (e.g., using robfig/cron).

If additional features are needed, extend Filebeat’s codebase.

Sample Logrotate Configuration

/var/log/xxxx/xxxxx.log {
  su www-data www-data
  missingok
  notifempty
  size 1G
  copytruncate
}

Conclusion

The presented design offers a simple, extensible approach to Kubernetes log collection. Teams should adapt the architecture to their specific requirements and scale.