Design and Implementation of 58 Private Cloud Platform Using Container Technology

58 Private Cloud is a business‑instance management platform built on container technology, offering on‑demand scaling, second‑level elasticity, a user‑friendly UI, and standardized testing and release processes to free developers from environment configuration and focus on business logic.

Why use container technology

Before containerization, the team faced low resource utilization, mixed‑deployment interference, slow scaling, inconsistent code across environments, and a lack of stable offline test environments, all of which hindered efficient development and operations.

Overall Architecture

The platform consists of several layers:

Infrastructure layer: servers, storage, network.

Container layer: Kubernetes for scheduling and management, an agent on each host for resource monitoring, and IPAM for network IP management.

Resource management layer: container management, scaling, rollback, release, quota and resource‑pool management.

Application layer: runs user‑submitted business instances in any programming language.

Basic components: service discovery, image registry, log center, monitoring center.

Unified portal: visual UI for standardized workflow and dynamic resource management.

Core Module Design

The platform focuses on five core modules: container management, logging collection, network model, monitoring & alerting, and image repository.

Container Management

After evaluating Docker‑based platforms (Swarm, Mesos‑Marathon, Kubernetes), the team selected Kubernetes for its dedicated container scheduling, active community, and rich ecosystem.

Network Model

Six Docker networking modes were compared and benchmarked (Iperf for bandwidth, Qperf for latency). The results showed host and bridge modes closest to native performance. The platform chose a bridge+VLAN scheme for its good performance, simple integration, easy debugging, and compatibility with existing RPC‑based services.

Because VLAN IDs are limited, an IPAM module was built (based on the open‑source Shrike project) that stores allocatable subnets in etcd; containers obtain IPs from this service and return them on termination. Custom patches were also applied to Kubernetes to support CNM.

Image Repository

Various storage back‑ends were evaluated. Although Ceph performed best, the team selected HDFS as the image repository backend due to existing expertise and operational support, with a future plan to migrate to an internally developed object storage solution.

Logging System

To address log loss in containers, a centralized logging solution was built: agents on hosts collect logs based on environment variables, forward them via Flume to Kafka, then Kafka fans out to a search service for portal queries and to HDFS for historical analysis.

Monitoring & Alerting

After evaluating several open‑source options, the team adopted WMonitor for container monitoring because it integrates host monitoring and alert logic, reducing development effort. Heapster + InfluxDB + Grafana remains an alternative for smaller deployments.

Conclusion

The article shares the technical exploration of container adoption within 58's architecture, highlighting key design decisions and trade‑offs. Future posts will dive deeper into each module’s implementation details and encountered challenges.