Design and Implementation of Container Network Architecture for Tongcheng Travel PaaS Platform

Background Tongcheng Travel's PaaS platform was built starting in 2014 and has grown from a few hundred containers to tens of thousands, evolving from manual scheduling to automatic scaling and undergoing three major version upgrades.

Content Overview The article introduces the platform's CI/CD practices, evaluates Docker network options, and describes the network redesign process, covering the system's initial state, Docker network customization, IP persistence solutions, and network monitoring.

1. System Initial State The platform originally used Docker Swarm with the Host network mode, which shares the host's IP stack. This simple setup worked for a small number of stateless containers but caused port conflicts and lacked IP isolation as the number of containers and services (e.g., Redis, Hadoop, databases) grew.

2. Public‑IP Based Network Customization To give each container an independent IP, several network schemes were evaluated:

Tunnel solutions (Weave, Open vSwitch, Flannel) – discarded due to performance loss and operational complexity.

Routing solutions – Calico (BGP‑based routing with fine‑grained ACLs) and Macvlan (layer‑2 isolation). Calico was chosen for its support of Docker CNM and Kubernetes CNI models, high performance, and rich policy features.

Calico implements a distributed virtual router on each node, propagates workload routes via BGP, and provides iptables‑based network policies for multi‑tenant isolation.

3. VLAN‑Based Solution – Contiv Contiv netplugin (Cisco open‑source) offers L2 (VLAN), L3 (BGP), and overlay (VXLAN) networking, integrates with SDN solutions, and supports both CNM and CNI. It enables multi‑tenant networking, policy enforcement, and IP persistence.

The CNM model defines Sandbox (container network namespace), Endpoint (virtual NIC), and Network (shared virtual bridge). Contiv’s REST API is used to allocate an IP before container creation; the allocated IP is then bound to the container’s Endpoint, achieving persistent IP addresses.

Network Monitoring OVS sFlow is used to capture host‑level traffic, a custom sFlow collector parses the data and stores key metrics in Redis. An ELK cluster periodically pulls data from Redis for aggregation and analysis, and the front‑end queries ELK APIs to display monitoring dashboards.

Through these changes, the platform migrated from Swarm to Mesos + Marathon, automated scaling, and continuously optimized the network stack.

Three years of continuous iteration have resulted in a robust, scalable PaaS solution that meets higher performance and reliability requirements.