Master Kubernetes Logging: Best Practices and Essential Tools

What is Kubernetes logging?

Kubernetes logging refers to the capture, storage, and management of log data generated by cluster components (API server, kubelet, container runtimes) and the applications running inside pods. The logs provide essential information for debugging, performance monitoring, and security auditing.

How logging works in Kubernetes

Each component writes logs to stdout/stderr or to log files on the node. Log‑collector agents run as DaemonSet pods (e.g., Fluentd, Logstash, Fluent Bit). These agents read container logs, optionally enrich or transform them, and forward the records to a central storage or analysis system such as Elasticsearch, Loki, or a cloud logging service.

Common log‑collection agents

Fluentd – a flexible, open‑source data collector that supports over 500 output plugins (Elasticsearch, Splunk, Kafka, etc.).

Logstash – part of the Elastic Stack; parses, filters, and ships logs to Elasticsearch or other destinations.

Fluent Bit – a lightweight Fluentd‑compatible agent designed for low‑resource environments.

Logging best practices

Use a centralized logging solution – Store logs in a single system (e.g., Elasticsearch, Loki, Splunk) to simplify search and correlation.

Implement log rotation – Configure size limits, retention count, and rotation frequency to prevent disks from filling.

Adopt structured logging – Emit logs in JSON or another structured format to enable efficient filtering and analysis.

Avoid logging sensitive data – Do not write passwords, API keys, or other credentials; use environment variables or secret managers instead.

Include contextual fields – Add timestamps, pod name, namespace, node name, request ID, etc., to each log entry.

Mask privacy‑sensitive fields – Replace PII or secret values with placeholders such as "****" before shipping logs.

Use appropriate log levels – Classify messages with DEBUG, INFO, WARNING, ERROR, or CRITICAL to prioritize alerts.

Stream logs in real time – Forward logs continuously to the central system to enable near‑real‑time monitoring and alerting.

Additional CLI tools for log collection

Kubetail – Aggregates logs from multiple pods and streams them to the terminal.

Stern – Provides color‑coded, multi‑pod tailing with regex‑based filtering.

Kail – Supports raw and JSON output and can filter by label or namespace.

Logcli – Part of the Loki ecosystem; queries logs stored in Loki using label, time‑range, and level filters.

kubectl log commands

Tail logs from a specific pod: kubectl logs -f <pod-name> Tail logs from pods matching a label selector: kubectl logs -f -l <label-selector> Tail logs from all pods in a namespace: kubectl logs -f -n <namespace> Use a wildcard pattern to tail multiple pods: kubectl logs -f <pod-name-pattern*> Include timestamps in the output: kubectl logs -f --timestamps <pod-name> Stop tailing with Ctrl+C.

Viewing control‑plane component logs

Replace the placeholder with the actual pod name of the component you want to inspect.

API server:

kubectl logs -f -n kube-system <api-server-pod-name>

Controller manager:

kubectl logs -f -n kube-system <controller-manager-pod-name>

Scheduler:

kubectl logs -f -n kube-system <scheduler-pod-name>

etcd cluster:

kubectl logs -f -n kube-system <etcd-pod-name>