Mastering Kubernetes Log Collection: Strategies, Tools, and Best Practices

Challenges of Log Collection in Kubernetes

Kubernetes abstracts the underlying compute resources and provides fine‑grained scheduling, which makes log collection more complex than on traditional VMs or physical servers. Collectors must handle short‑lived jobs, high container density per node, and declarative deployment of logging configurations.

Collection Modes

Passive Collection (Sidecar, DaemonSet)

A DaemonSet runs a single log agent on each node and gathers logs from all containers on that node. It has low per‑node resource consumption but limited scalability and tenant isolation, making it suitable for small‑to‑medium clusters.

A Sidecar deploys a dedicated log agent container alongside each application pod. This consumes more resources per pod but provides strong isolation and flexible configuration, which is ideal for large or PaaS‑style clusters.

Active Collection (Docker Engine LogDriver, Direct SDK Writes)

Docker Engine LogDriver forwards container stdout/stderr to a remote storage backend. It offers minimal customisation, low resource isolation and is generally discouraged for production workloads.

Direct SDK writes embed a logging SDK in the application code, sending logs straight to the backend without writing to files. This incurs the lowest runtime overhead but tightly couples business logic with the logging implementation and is recommended only for extremely high‑volume scenarios.

Log Output Options

Containers can emit logs via stdout/stderr or write to files . While Docker recommends stdout, the path adds processing overhead (e.g., JSON LogDriver serialization) and mixes different log types into a single stream, which hampers categorisation and increases CPU usage (≈1 CPU core per 100 k lines/s in benchmark tests). File output allows configurable rotation, compression, sync/async writes, and clear separation of access, error, trace, and other log categories, making it the preferred choice for production workloads.

CICD Integration via Logging Operator

The logging solution is exposed as a Kubernetes Custom Resource Definition (CRD) named AliyunLogConfig , managed by a Logging Operator. Deploying an AliyunLogConfig object declaratively integrates log collection into CI/CD pipelines without custom scripting.

Installation

The components are packaged as a Helm chart that includes:

Logtail DaemonSet

AliyunLogConfig CRD definition

CRD controller

For Alibaba Cloud clusters the chart can be installed automatically during cluster creation; for self‑managed clusters it can be installed manually, for example:

helm repo add aliyun-log https://aliyun-log-charts.example.com
helm install aliyun-log aliyun-log/aliyun-log

After installation the Logtail agents run in the cluster but do not collect any logs until a configuration is applied.

Configuration Methods

Two approaches are supported:

Environment variables : Set variables such as ALIYUN_LOG_ENDPOINT and ALIYUN_LOG_PROJECT in the target pod. This method is simple but limited to basic collection and requires recreating pods for any change.

CRD (AliyunLogConfig) : Declare a resource that specifies source paths, log types, parsing rules, filters, and destination endpoints. The CRD supports full CRUD semantics, advanced parsing, black‑/white‑list filtering, and can be version‑controlled alongside other Kubernetes manifests.

Recommended Configuration Strategy

Consolidate similar log sources into a single AliyunLogConfig to reduce the number of DaemonSet resources.

Allocate dedicated Sidecar agents for critical services that require high reliability and isolation.

Prefer CRD‑based configurations for flexibility, manageability, and alignment with Kubernetes best practices.

Use Sidecar agents in ultra‑large clusters where each business team needs independent log‑tail settings.

Practical Scenarios

Small/Medium Clusters (≤500 applications, ≤1000 nodes)

Deploy the Logtail DaemonSet cluster‑wide for the majority of workloads. Deploy Sidecar agents only for core services such as order or transaction systems that demand higher reliability.

Large/Ultra‑Large Clusters (≥1000 applications and nodes)

Collect system component logs (kernel, kube‑system, etc.) with the DaemonSet. Collect business‑level logs with Sidecar agents, giving each team independent configuration and resource allocation.

Key Takeaways

Docker Engine LogDriver is generally unsuitable for production.

Direct SDK writes are only recommended for extremely high‑volume log streams.

DaemonSet is the default choice for small‑to‑medium clusters.

Sidecar is recommended for large or ultra‑large clusters where per‑application isolation is required.