From Java Monolith to Serverless Kubernetes: Building a Custom Container Orchestration System

Background

Initially a Java monolith runs on a single physical server. As traffic grows the hardware is repeatedly upgraded (e.g., from 64 CPU 256 GB RAM to 160 CPU 1920 GB RAM) until a single machine can no longer satisfy the load. The monolith is refactored into multiple micro‑services using middleware (HSF, TDDL, Tair, Diamond, MetaQ). After the split the number of servers increases, leading to heterogeneous hardware, OS versions and operational complexity. Virtual machines hide these differences but introduce noticeable performance overhead.

Docker is adopted because it leverages Linux cgroups for lightweight isolation and enables image‑based CI/CD pipelines. When the container count reaches thousands, new challenges appear: automated scheduling, resource allocation, and inter‑container networking at scale.

Design of a Container Orchestration System

The system follows a master‑worker architecture.

Master node components kube-apiserver: exposes a RESTful management API used by administrators (via kubectl) and by worker agents ( kubelet). kubectl: command‑line client for cluster operators to create, update, and delete workloads. kube-scheduler: watches pending workload objects, reads resource reports from workers, and selects the most suitable node based on CPU and memory availability. etcd: strongly consistent key‑value store that persists cluster state (node inventories, pod specifications, etc.). kube-controller-manager: runs a set of controllers (node controller, replica controller, endpoint controller, etc.) that continuously reconcile the desired state stored in etcd with the actual state of the cluster.

Worker node components kubelet: runs on each worker, periodically reports node capacity, allocatable resources, and the status of all containers to the master via the API server. kube-proxy: watches Service objects and updates network forwarding rules (iptables or IPVS) so that container IPs are reachable across nodes.

Typical workflow

An operator runs a command such as:

kubectl run user-center --image=repo/taoche/user-center:2.0 --replicas=1000

The API server stores the desired pod objects in etcd. The scheduler reads the pending pods, queries the latest node resource reports from kubelet, and assigns each pod to a worker that has sufficient CPU and memory. The scheduler then instructs the selected workers to create the containers. If a node fails, the node controller marks it unschedulable, the replica controller detects the loss of pods, and the scheduler creates replacement pods on healthy nodes.

Networking model

Each pod receives a unique IP address from a cluster‑wide CIDR; containers communicate directly via these IPs.

Worker nodes also have unique IPs; routing between pod IPs on different nodes is handled by kube-proxy using iptables or IPVS rules.

When pod IPs are added, removed, or changed, kube-proxy automatically updates the forwarding rules to keep connectivity consistent.

Service abstraction

To hide volatile pod IPs, a Service object provides a stable virtual IP (VIP) or DNS name. The Service forwards traffic to the set of backing pods, and an internal DNS server resolves the Service name to its VIP, enabling applications to address services by name rather than by individual pod IPs.

Optional plugins

Cluster‑internal DNS for Service name resolution.

Web UI (Dashboard) for visual management of resources.

Metrics server or custom exporters for resource monitoring.

Centralized logging agents for aggregating container logs.

Summary of core components

Master components : kube-apiserver, kube-scheduler, etcd, kube-controller-manager.

Node components : kubelet, kube-proxy.

Plugins : DNS, web UI, resource monitoring, log aggregation.

Future direction

After building a production‑grade orchestration platform, organizations often evaluate serverless‑style offerings (e.g., Alibaba Cloud Serverless Kubernetes) to further reduce operational overhead, but the fundamental architecture described above remains the basis for any Kubernetes‑compatible system.