How 360 HULK Overcomes Container Log Collection Challenges with Log‑Controller

Speaker Introduction

Wang Xigang , Operations Development Engineer at Qihoo 360 Web Platform Department, has extensive experience in PaaS design, Kubernetes, Docker, and leads the development of the HULK container cloud platform.

Background

The HULK cloud platform supplies core services such as LVS load balancing, container services, S3/Ceph storage, and message queues to over 90% of the company's business. Container services are a key product, and efficient log collection is essential for observability.

Log Collection Challenges

Issues Caused by Container Characteristics

Containers write logs to standard output, but many applications also write to local disks, making it hard to capture those logs.

Elastic scaling means Pods can be scheduled on any node, preventing static configuration of log paths.

Deficiencies of Existing Log Collectors

They lack dynamic awareness of configuration changes, requiring manual updates after deployments.

Log loss can occur when a Pod is terminated and its log files are deleted before collection.

Without clear source tagging, it is difficult to identify which Pod generated a particular log entry.

Solution Overview

Addressing Container‑Specific Issues

A custom service called log‑controller was built with two main functions:

Dynamically detects changes in Kubernetes Nodes and Pods, generating per‑Node log‑collection configurations.

Periodically scans all Pods to keep configuration data consistent.

The service defines collection paths for both standard output (Docker Engine logs stored under /data/docker/containers/) and application logs mounted via emptyDir volumes.

Fixing Existing Collector Defects

log‑controller pushes generated Logstash configurations to the internal configuration center Qconf; Logstash agents on each Node pull and hot‑load these configs.

Logstash runs as a DaemonSet with the close_older setting (default 3600 s) to ensure logs from terminated Pods are still captured.

Logstash supports a custom message prefix ( custome_msg_prefix) to tag logs with their originating Pod, enabling precise source identification.

Architecture of Container‑Cloud Log Collection

The overall architecture consists of:

master : Kubernetes control plane components.

log‑controller : Dynamically generates Logstash configs.

Qconf : Internal configuration center.

Logstash : Deployed as a DaemonSet on every Node to collect logs.

Qbus2.0 : Kafka‑based message queue for log transport.

Elasticsearch and HDFS : Storage back‑ends for processed logs.

Log Collection Workflow

1. Users create an application (e.g., nginx) in the container service UI, specify the desired Pod replica count, and optionally enable log collection, which registers a topic in Qbus2.0.

2. The request is sent to the Kubernetes master, which schedules Pods onto suitable Nodes.

3. log‑controller detects the new Pods, retrieves the associated topic and log paths, generates Logstash configuration files, and pushes them to Qconf.

4. Logstash agents on each Node poll Qconf (default every 30 s), hot‑load the latest configs, and collect logs from the defined paths.

5. Collected logs are sent to Qbus2.0. They can either be stored directly in HDFS or, after optional cleaning, forwarded to Elasticsearch for indexing and visualization via Kibana.

Conclusion

The log‑controller and associated pipeline effectively resolve the dynamic, loss‑prone, and source‑identification challenges of container log collection on a large‑scale Kubernetes platform, providing a reliable foundation for further log analysis and monitoring.