Design and Architecture of Ping An Cloud Container Service Platform

The document introduces the motivation behind adopting container technology at Ping An Cloud and defines the overall goal of providing a self‑service, virtual‑machine‑like container platform for various financial subsidiaries with differing compute needs.

It outlines four main parts: platform positioning, design, architecture, and technical implementation.

Platform Positioning : The platform serves as an internal public cloud offering multiple compute resources (physical machines, VMs, containers) and extends existing cloud services (hosts, storage) with container capabilities.

Platform Design : Emphasizes multi‑tenant isolation, permission control, and seamless integration of different compute models. The design includes a user‑facing CaaS portal, a product overview diagram, and a conceptual model defining tenants, environments, hosts, applications, orchestration (Docker Compose), services, containers, and images.

Key Functionalities include:

Container environments: isolated per tenant with elastic resource scaling.

Application deployment via Docker Compose V1 with one‑click publishing.

Service management supporting port mapping, environment variables, volume mounts, and extensions like shell access and image building.

Image management with a marketplace, shared public repository, and private tenant repositories, supporting multi‑region distribution.

Architecture : Describes the overall system layout across data‑center zones, separating external cloud‑management zones (CaaS portal, image marketplace) from internal resource zones. Each zone runs Rancher‑based orchestration, MySQL Galera clusters for the portal, and custom Docker Server agents.

Technical Components :

Docker Server: Java‑based wrapper around the Remote Docker API.

Orchestration: Rancher for container lifecycle management.

Registry Mirror: Local registry per data center to reduce external traffic and improve security.

Configuration tool: Ansible wrapped as a RESTful service for host provisioning.

Container Networking : Covers three evolution stages—from basic bridge/host/network‑none models, through tunnel (IPSec, Fannel) and routing (Calico) approaches, to modern CNI standards. The platform uses bridge with port mapping for external access and IPSec tunnels for internal communication.

Container Storage : Chooses DeviceMapper on CentOS for production‑ready storage, leveraging LVM and cloud disks, and supports volume mounts for application data.

Logging : Provides three log layers—platform logs (local + ELK), container runtime logs (local + ELK), and application logs (user‑managed with directory mounts).

Monitoring : Integrates existing cloud monitoring with custom scripts for host and container metrics (CPU, memory, network, storage) and middleware monitoring (WebLogic, Tomcat, Nginx) via Zabbix and Open‑Falcon, viewable in the portal.

Image Management : Evolves from single‑node to multi‑node and cross‑region distributed setups using Distribution, LVS + Keepalived, and DNAS storage, with synchronization mechanisms (active push/pull and event notifications).

The article concludes with a Q&A addressing cross‑region communication, DeviceMapper issues, permission handling, high‑availability of registries, and security considerations, emphasizing internal AD authentication and tenant isolation.