How OPPO Secures Docker at Scale: Deep Dive into Container Security Practices

This article is based on Liu Zhanlu's 2019 GOPS Global Operations Conference talk titled "Container Era: Deep Docker Security Practices for Tens of Thousands of Servers".

1. Docker Ecosystem and Architecture

The container ecosystem includes Docker engine, images, registries, and orchestration platforms such as Kubernetes. Security must address the entire stack, not just the container runtime.

2. Security Issues and Countermeasures

2.1 Host and Docker Daemon Security

Risks include running the daemon with root privileges and exposing Docker Remote API to the Internet.

Use TLS/HTTPS for daemon communication.

Place a secure proxy in front of the daemon.

Restrict API access with ACLs and enable TLS authentication.

2.2 Image Security

Compromised images can introduce malware, mining code, or vulnerable libraries. Scanning tools such as Clair, Anchore, and Dockerscan are used to detect known CVEs and malicious content.

2.3 Runtime Security

Runtime threats include container escape (e.g., runC vulnerability), resource exhaustion, and inter‑container DDoS. Mitigations rely on kernel security features such as seccomp, capabilities, SELinux/AppArmor, and traffic control.

2.4 Ecosystem Security

Kubernetes, as the dominant orchestration platform, also faces security challenges. Best practices include minimal privileges, regular patching, audit logging, and securing API server communication.

3. OPPO’s Container Security Practices

OPPO secures the entire container lifecycle—from image build to deployment, runtime, and destruction.

3.1 Secure Image Build

Use trusted base images.

Run services with non‑root users.

Prefer

COPY

over

ADD

and avoid storing secrets in Dockerfiles.

3.2 Registry Hardening

All registry access goes through a proxy that enforces authentication, authorization, encrypted transport, and audit logging.

3.3 Deep Image Scanning

Images are pulled, their Dockerfile history is analyzed, and each layer is scanned for malicious binaries, vulnerable libraries, and leaked credentials using CVE databases, YARA rules, and custom signature sets.

3.4 Host Hardening

Apply AppArmor/SELinux policies.

Use seccomp and capability restrictions.

Audit Docker daemon logs and configuration.

3.5 Log Collection and Traffic Analysis

Unified log agents forward container logs to a big‑data platform for anomaly detection using rule‑based and machine‑learning methods. Network traffic on both the Docker bridge and host interfaces is monitored to detect inter‑container attacks.

3.6 Process Monitoring and Intrusion Detection

Processes inside containers are mapped to their container IDs, and abnormal behavior (e.g., reverse shells) triggers alerts. Critical binaries such as

docker-runc

and

dockerd

are continuously integrity‑checked.

4. Outlook for Container Security

Future challenges include securing automated orchestration pipelines, managing secret lifecycle, handling container drift, and improving kernel isolation to reduce reliance on host‑level defenses.