Operator‑Based Log Collection and the Evolution of Loggie in Cloud‑Native Environments

NetEase began exploring cloud‑native technologies in 2015, and its log platform evolved from host‑based collection to container‑aware solutions to support massive cloud‑native migrations across the company.

Operator‑based log collection – Three common approaches are described: (1) collecting only standard output, (2) globally mounting log paths via a DaemonSet agent, and (3) deploying a sidecar agent per pod. Each method’s advantages and drawbacks (e.g., configurability, resource consumption, invasiveness) are analyzed.

Challenges at large scale – As more services moved to Kubernetes, the original Filebeat‑centric pipeline showed stability, performance, and operational issues: single‑queue bottlenecks, difficulty handling multiple Kafka clusters, limited observability, and high maintenance overhead.

Introducing Loggie – To address these pain points, NetEase built a custom, Golang‑based log‑collection agent called Loggie. It offers a one‑stack solution that can act as both collector and aggregator, supports hot‑plug pipelines, provides rich metrics via Prometheus, REST, and Kafka, and integrates tightly with Kubernetes CRDs for configuration.

Key features include: a plug‑in architecture for easy extension, sidecar and DaemonSet deployment options, built‑in log alerting via interceptors, support for Kubernetes events, and production‑grade capabilities such as QPS limiting, disk‑usage protection, and FD management.

Performance evaluation – Benchmarks comparing Loggie with Filebeat show Loggie consumes only a quarter of the CPU while delivering 1.6‑2.6× higher throughput, raising the practical ceiling from ~80 MB/s to over 200 MB/s.

Open‑source roadmap – Loggie has been open‑sourced (https://github.com/loggie-io/loggie/) and the project roadmap invites community contributions to further enhance cloud‑native logging capabilities.