How to Build a Scalable Kubernetes Logging System with S6 and Filebeat

Docker logs are divided into engine logs and container logs. Engine logs go to system logs, while container logs are the output of applications inside containers, accessible via

docker logs

and stored in

/var/lib/docker/containers/<container_id>/<container_id>-json.log

, which is unsuitable for production.

Problems with default logging include unlimited file size leading to disk exhaustion, Docker daemon becoming a bottleneck, and blocking of commands like

docker logs -f

and

docker ps

when logs are large.

Docker offers logging drivers; however, they still rely on the daemon, so performance remains limited. Benchmarks show syslog at 14.9 MB/s and json‑file at 37.9 MB/s.

Using an S6‑based image, logs can be redirected directly to host files with automatic rotation, bypassing the daemon.

k8s Logging

Kubernetes logging is categorized into three levels: pod (application), node, and cluster.

Pod‑level logs are accessed with

kubectl logs pod-name -n namespace

.

Node‑level logs are managed by configuring container log‑drivers together with logrotate.

Cluster‑level logging can be implemented via node agents (DaemonSet) or sidecar containers. Node agents collect logs on each node with minimal resource usage. Sidecars can either stream logs to stdout (creating duplicate files) or run a dedicated log collector such as Logstash or Filebeat, though the latter consumes more resources and hides logs from

kubectl logs

.

Logging Architecture

A unified logging system can be built using a node‑agent DaemonSet that runs on every node. All application containers use an S6 base image, redirecting logs to host directories like

/data/logs/namespace/appname/podname/log/xxxx.log

. The log‑agent contains Filebeat and logrotate; Filebeat ships logs to Kafka, which forwards them to Elasticsearch for storage and Kibana for querying. Logstash creates indices and consumes Kafka messages.

Key challenges include dynamically updating Filebeat configurations for new applications, ensuring log rotation for all files, and extending Filebeat with custom features.

Practical Solutions

Dynamic Filebeat config updates can be achieved with

fsnotify

to watch log directories and render templates.

Log rotation can be scheduled with

cron

jobs using the

logrotate

utility, e.g.:

<code>/var/log/xxxx/xxxxx.log {
    su www-data www-data
    missingok
    notifempty
    size 1G
    copytruncate
}
</code>

Custom Filebeat development references are available online.

Conclusion

The article provides a simple approach to Kubernetes log collection; implementations should be adapted to specific company requirements.