Applying Mesos and Docker Containerization in 360 Commercial Advertising System

The talk by senior engineer Li Dong (April 14) explains how Mesos and container technology are integrated into 360's commercial advertising system to address key business pain points such as data‑center migration, fault recovery, operating‑system inconsistencies, production and test environment differences, low scalability, and poor resource utilization.

Containers are chosen for their standardization benefits; Docker’s “container” metaphor provides a uniform size and lifecycle, enabling consistent deployment across environments. Docker standardization is achieved through Dockerfiles, the docker run command, and immutable image layers that ensure idempotent deployments.

During containerization, the team avoids unnecessary services like SSH inside containers, and logs are collected via Elasticsearch, Graylog, and Grafana, ensuring observability without direct container access.

Network performance is evaluated using Host‑only and Bridge modes, with Calico employed for dedicated external IPs or network isolation when needed.

Image storage is handled by Harbor backed by S3, providing high reliability, authentication, and future CDN distribution.

Data persistence options include CephFS, MySQL, Kafka, Aerospike, and Redis, allowing both stateless and stateful workloads within containers.

Service registration and discovery rely on mesos‑dns and Marathon, with Zookeeper ensuring high‑availability for master nodes; the architecture also supports dynamic scaling and fault recovery.

Mesos is adopted for its high resource utilization, modularity, and extensibility. Its architecture consists of a Zookeeper‑backed master, agents, and frameworks (e.g., Marathon, Flink) that handle task scheduling.

The implementation uses roughly 20,000 lines of C++ code, with fault‑tolerance provided by Zookeeper and modular frameworks.

Mesos master failover is managed via Zookeeper leader election, allowing a switchover in about ten seconds.

The ecosystem includes Marathon (long‑running jobs, service discovery, health checks, LB), Marathon‑lb (HAProxy‑based load balancing), and Mesos‑DNS for name resolution.

Comparisons with YARN and Kubernetes highlight Mesos’s flexible resource scheduling versus YARN’s CPU/Memory focus and Kubernetes’s container‑centric automation.

Practical applications at 360 include containerizing Storm clusters, image services (PHP7), and various other services (Web, Aerospike, Kafka, Redis), with CI/CD pipelines driven by GitLab Runner to build Docker images and update Marathon deployments automatically.

Future plans involve expanding CephFS usage, exploring Calico for advanced networking, and integrating real‑time machine‑learning workloads on the Mesos platform.