Migrating Test Environment from VMs to Kubernetes: Architecture, Steps, and Customizations

The Youzan PaaS team began allocating testing resources in July 2017. As test personnel joined, a series of testing‑related artifacts such as test environments, engineering processes, and workflows emerged. Initially the test environment was deployed on virtual machines (VMs). Starting in July 2018 the team decided to migrate the test environment to Kubernetes (K8S) for several reasons.

Why migrate?

Continuous delivery (CD) system only supports business products, not PaaS products, which have diverse languages, deployment complexities, and niche use cases. Therefore the PaaS test environment must be built by testers themselves.

Cost : Deploying >40 components of >15 PaaS products on VMs would require 40+ machines, potentially 70‑100 machines when accounting for multi‑node, master‑slave, and multi‑datacenter scenarios. This leads to high hardware cost and management overhead.

Deployment effort : VM‑based deployment needs extensive shell scripts in Jenkins jobs, SSH key distribution to every target machine, and manual maintenance of scripts. Moving to K8S allows all build, package, and deployment logic to live in Dockerfiles stored in GitLab, simplifying maintenance.

Trend : Docker and Kubernetes have become industry standards; adopting K8S keeps the testing team aligned with modern cloud‑native practices.

Overall architecture

The new architecture consists of four core components:

GitLab – source code repository.

Jenkins – continuous integration server that pulls code from GitLab, builds Docker images, and pushes them to Harbor.

Harbor – private Docker image registry.

Kubernetes – container orchestration platform that replaces VMs for application deployment.

Step‑by‑step migration

1. Integrate Kubernetes with Jenkins

Jenkins provides a Kubernetes plugin. After installing the plugin, configure a new cloud under System → Configure System → Cloud with the cluster address and certificates.

Example pipeline script (copy the following into a Pipeline job):

podTemplate(label: 'mypod', cloud: 'kubernetes', containers: [
    containerTemplate(
        name: 'jnlp',
        image: 'www.harbor.com/yzcontainer/yz-centos-jnlp-slave:latest',
        alwaysPullImage: true,
        args: '${computer.jnlpmac} ${computer.name}',
        env:[
            'name':'application_standard_env',
            'value':'daily'
        ]
    )
],
volumes: [
    hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock')
]){
    node('mypod') {
        stage('test') {
            container('jnlp') {
                sh """
                # your test commands here
                """
            }
        }
    }
}

2. Create a Namespace

Namespaces isolate resources. Create one via command line: kubectl create namespace <namespace-name> Or define a YAML file ( my-namespace.yaml) and apply it:

apiVersion: v1
kind: Namespace
metadata:
  name: <namespace-name>

Apply with:

kubectl create -f my-namespace.yaml

3. Create a Deployment

A Deployment provides declarative updates for Pods. Example deployment.yaml (trimmed for brevity):

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: <deploy-name>
  labels:
    run: <custom-label>
  namespace: <namespace>
spec:
  replicas: 1
  selector:
    matchLabels:
      run: <custom-label>
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        run: <custom-label>
    spec:
      containers:
      - name: <container-name>
        image: <image-name>
        imagePullPolicy: Always
        ports:
        - containerPort: 80

Create or apply with:

kubectl create -f deployment.yaml   # one‑time creation
kubectl apply -f deployment.yaml    # idempotent updates

Check status: kubectl get pod -n <namespace> -o wide If the pod is Running, the deployment succeeded; otherwise inspect logs with: kubectl logs -f <pod-name> -n <namespace> To force a redeploy when the image tag does not change, delete the pod:

kubectl delete pod <pod-name> -n <namespace>

4. Create a Service

Expose the Deployment via a Service. Example Service YAML:

kind: Service
apiVersion: v1
metadata:
  name: my-service
  namespace: <namespace>
spec:
  selector:
    app: MyApp
  ports:
  - protocol: TCP
    port: 2379
    targetPort: 2379
  type: NodePort

Or expose directly with kubectl:

kubectl expose deploy/etcd --port=2379 --target-port=2379 --name=etcd --type=NodePort -n <namespace>

Verify with:

kubectl get svc -n <namespace>

5. Create an Ingress

Ingress adds L7 routing capabilities. Example ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: yz7-ingress
  namespace: yz7
spec:
  rules:
  - host: yz7-test-control.s.yz.com
    http:
      paths:
      - backend:
          serviceName: yz7-cluster
          servicePort: 8888
        path: /

Create and view:

kubectl create -f ingress.yaml
kubectl get ing -n yz7 -o wide

Customizations

NodeName

Force a pod onto a specific node by adding nodeName: <node-ip> in the pod spec.

SecurityContext

Enable privileged network commands (e.g., tc, iptables) by adding capabilities:

securityContext:
  capabilities:
    add:
    - NET_ADMIN

Volumes

Persist data or share files between containers using hostPath volumes:

volumeMounts:
- mountPath: /data/etcd
  name: etcd-volume
volumes:
- name: etcd-volume
  hostPath:
    path: /data/etcd
    type: Directory

DNS

Access services via DNS names instead of IPs. A normal Service resolves to serviceName.namespace.svc.cluster.local. A headless Service (clusterIP: None) resolves to individual pod IPs.

Conclusion

By migrating all Youzan PaaS products from VM‑based test environments to a single Kubernetes cluster, the team achieved faster integration cycles, lower cost, and improved isolation. The migration also opened the door to advanced cloud‑native features such as node affinity, security contexts, persistent volumes, and ingress routing, while acknowledging that Kubernetes is a vast ecosystem that still requires continuous learning.