Mastering Kubernetes Logging: Overcoming Real‑World Challenges

Introduction

The author, responsible for Alibaba Cloud Log Service's data collection client (logtail), explains that the client is deployed at million‑scale across the company, ingesting petabytes of data daily and surviving multiple high‑traffic events like Double 11 and Double 12.

As Kubernetes (K8s) evolves, developers building K8s logging systems encounter increasingly complex problems. Drawing on years of hands‑on experience, the author analyzes the difficulties of constructing a K8s logging system and aims to provide practical guidance.

Why a Logging System Is Essential

In production troubleshooting, the typical workflow is: discover an issue via metrics, locate the problematic module using traces, and finally pinpoint the root cause through logs that contain errors, key variables, and execution paths. Therefore, logs are an indispensable part of any incident‑resolution process.

Evolution of Alibaba’s Logging Architecture

Single‑machine era: Applications ran on standalone servers; logs were primarily used for debugging and were analyzed with simple Linux tools like grep .

Distributed era (Feitian 5K project, 2013): To break the scalability bottleneck of single machines, Alibaba launched the Feitian 5K project, moving services to a distributed architecture. Centralized logging, tracing, and monitoring systems were built to collect logs, metrics, and traces from all services.

Container era: Recent years have seen a shift to containerization, full adoption of Kubernetes, and serverless workloads. Log volume and variety exploded, driving the need for a unified, digital, and intelligent log platform.

Observability in Cloud‑Native Environments

According to the CNCF definition, cloud‑native systems are built with containers, service meshes, micro‑services, immutable infrastructure, and declarative APIs, aiming for elasticity, fault tolerance, manageability, observability, and loose coupling. Observability’s ultimate goal is to enable digital and intelligent operations across the entire organization, spanning DevOps, business, BI, audit, and security.

Alibaba has developed a wide range of tools around its logging platform, including real‑time log analysis, distributed tracing, monitoring, data enrichment, stream and batch processing, BI, and audit systems. The platform focuses on real‑time collection, cleaning, intelligent analysis, and seamless integration with downstream processing pipelines.

Key Challenges of Building a Logging System on Kubernetes

Complex log sources: Besides traditional VM logs, Kubernetes generates container stdout, container file logs, container events, and K8s events, all of which must be collected.

Highly dynamic environment: Pods can be created, scaled, or destroyed at any moment, making log files transient. Logs must be captured in real time and the collection agents must adapt to rapid topology changes.

Proliferation of log types: A typical request traverses CDN, Ingress, Service Mesh, and multiple Pods, producing system component logs, audit logs, service‑mesh logs, and more.

Changing business architecture: Micro‑service adoption increases inter‑service dependencies, making cross‑dimensional log correlation more difficult.

Integration difficulty: Logging must fit into CI/CD pipelines and align with Kubernetes’ declarative deployment model, yet many existing log solutions are standalone and costly to integrate.

Scale problems: Open‑source self‑built log systems work for early testing, but as log volume grows, issues such as tenant isolation, query latency, data reliability, and system availability emerge, especially during large‑scale events where concurrent queries can overload the platform.

Looking Ahead

The author promises a follow‑up series that will detail Alibaba’s concrete implementation of a Kubernetes logging system, covering architecture, component selection, deployment practices, and operational tips.