Meituan-Dianping Docker Container Platform: Architecture and Practices

This article, originally presented at Meituan-Dianping's technical salon "The Meituan Cloud You Don't Know" and published in the January 2017 issue of Programmer , describes the design and implementation of Meituan-Dianping's Docker container management platform.

The platform, launched in 2015 on Meituan Cloud infrastructure, serves dozens of business units (food delivery, hotel, cinema, etc.) and handles hundreds of online services with over 45 billion daily requests, covering Web, database, cache, and message queue workloads.

Key motivations for building the platform include the need for rapid, automated scaling beyond the capabilities of traditional virtual machines, and the reduction of high IT costs caused by over‑provisioning of VMs for traffic spikes.

Architecturally, the platform follows a four‑layer model: Business Layer (users of containers), PaaS Layer (HTTP APIs for orchestration, scaling, monitoring), IaaS Control Layer (scheduling, networking, authentication, image registry), and Host Resource Layer (Docker hosts with HostServer, monitoring agents, volume management, OVS networking, cgroup control).

Core components:

API : Unified interface for creating and managing both VMs and containers, simplifying migration and reducing launch latency to under 3 seconds for containers.

Host‑SRV : Host‑side process manager handling image pull, container lifecycle, and volume management via Unix socket communication with Docker Daemon.

Image Repository : Combines Docker Registry (mirrored Hub) and an extended OpenStack Glance service to store Docker images, enabling unified management of VM and container images.

Network : High‑performance, high‑elasticity network built on OVS‑DPDK for data plane and a custom controller for the control plane; MosBridge driver integrates Meituan Cloud networking with Docker Libnetwork.

Storage : LVM‑backed volume solution that enforces strict capacity limits per container, providing isolation and online expansion capabilities.

Monitoring : Mos‑Docker‑Agent collects metrics from proc and cgroup interfaces and reports to existing monitoring systems (Zabbix, Falcon, CAT) with low overhead.

To support micro‑service architectures, the platform introduces the concept of a set (similar to Kubernetes Pods), grouping a BusyBox management container with one or more business containers that share network, volume, and IPC resources. Sets are described by JSON configurations specifying container index, image, and resource options.

For Docker stability and feature gaps, Meituan‑Dianping maintains a custom branch called MosDocker (based on Docker 1.11) that adds MosBridge networking, persistent cgroup settings, and sub‑image save optimizations.

Operational results after a year of deployment include:

Seconds‑level scaling for traffic spikes, reducing expansion time from hours to seconds.

Significant hardware and operational cost savings by eliminating over‑provisioned VMs.

Live resource adjustments (CPU, memory, disk) for stateful services without container restarts.

The article concludes that Docker, combined with traditional VMs, offers a complementary approach to meet diverse cloud computing scenarios.