Re‑engineering a Legacy Ops Platform with Kubernetes and Docker

Background

The company’s backend micro‑service architecture had grown to over a thousand Java modules, managed by a simple Jenkins‑driven deployment process that copied JARs, config files, and start scripts to bare‑metal servers. As the number of projects and request volume increased, the platform showed critical shortcomings: lack of clear resource accounting and isolation, difficulty managing multiple JDK versions and custom Java agents, and the need to reclaim server access from developers.

Technical Selection

To address these issues, the team chose mature open‑source container technologies. Docker and Kubernetes were selected as core components, with Kubernetes deployed directly on bare‑metal servers (no VM layer). For storage, most services are stateless and require no distributed storage; the few that do use an existing MFS cluster mounted on the host and then into containers. Docker’s default OverlayFS 2 is used for local volumes, running on CentOS with DeviceMapper in direct‑LVM mode. Logging is handled by an existing ELK stack, with container logs collected via mounted data disks to avoid performance impact of console logging. Monitoring moved from Zabbix to Prometheus + Grafana, using kube-state-metrics for richer metrics. For networking, the team adopted kube‑router , a three‑layer routing and BGP‑based CNI, which offers better performance than Flannel/VXLAN and is lighter than Calico, while also supporting IPVS‑based Service Proxy.

Business Implementation

The migration goal was to let developers move projects to Kubernetes without code changes. A unified Dockerfile template handles different JDK requirements. A custom build worker, based on Kubernetes Jobs and Docker‑in‑Docker (dind), pulls compiled artifacts from Jenkins, builds images, and pushes them, enabling both VM‑based and container‑based deployments from the same artifact.

Deployment configuration is split into static resource definitions (managed in the ops platform) and dynamic version/instance changes. The platform generates Deployment and Service objects from these parameters and invokes the Kubernetes API. ConfigMaps store static configuration files, and when needed, Apollo environment variables are injected. The UI provides pod lists, debugging containers, gray‑release controls, status monitoring, and a web‑shell, eliminating the need for developers to SSH into production servers.

Challenges Encountered

Pod‑to‑external communication : Since Dubbo services still run outside the cluster, static routes or BGP peering were added to expose the Pod IP range to external networks.

JVM memory vs. container limits : Containers need memory limits higher than the JVM Xmx setting (about 20% extra) to avoid OOMKill caused by Metaspace, PermSize, and off‑heap allocations.

Pod startup failures : Debug containers were introduced, allowing developers to launch a pod with the same spec but without health checks and with an overridden command, then attach via web‑shell for troubleshooting.

Future Work

Integrate lightweight debugging tools (e.g., Vim, Arthas) without inflating base images, possibly by launching side‑car debug containers.

Implement multi‑cluster scheduling to burst into public clouds when on‑prem resources are insufficient.

Enhance auto‑scaling beyond basic HPA by incorporating additional metrics for more precise scaling decisions.

Q&A

Q1: Are Pod networks fully reachable from external networks? A1: Yes, by adding static routes for the Pod IP range on the top‑level router or using BGP.

Q2: How is I/O isolation handled? A2: Currently there is no I/O isolation; Docker supports IOPS limits, but Kubernetes support is unclear.

Q3: How do Job and dind work together for image building? A3: dind mounts the host Docker client/socket inside a container, allowing the container to invoke Docker for builds; a Kubernetes Job schedules this worker.

Q4: Are ConfigMaps or third‑party tools used for configuration? A4: ConfigMaps are used for static files; newer projects use Ctrip’s Apollo via environment variables. SkyWalking is used for APM.

Q5: Why not use Macvlan/IPvlan? A5: They require VLAN configuration on switches and offer little performance gain over the simpler three‑layer routing solution.

Q6: How are multi‑line container logs collected? A6: Logs are written to files; Filebeat’s multiline support aggregates them, and they are stored in Elasticsearch for UI retrieval.

Q7: What storage is used? A7: Only a few projects need distributed storage; the existing MFS cluster is mounted on the host and then into containers.

Q8: Is Jenkins using pod templates for slaves? A8: No, Jenkins runs in a traditional master‑slave mode; the platform calls Jenkins APIs for deployment.

Q9: Are container images optimized for size? A9: Images range from 100‑300 MB; size optimization is secondary to stability and debugging convenience, and layered updates avoid full image pulls.

Illustrations