Step-by-Step Deployment of K3s with Kube‑OVN, Multus, VPN and Cloud‑Hypervisor for a Hybrid Container‑VM Environment

Architecture Overview

The diagram shows K8s abstracting physical servers and networks, while OpenVSwitch (Kube‑OVN) creates a multi‑tenant VPC where containers and VMs can communicate.

Experience Environment

Each student receives 18 containers, 1 VM, and 1 router that can be started, stopped, or reset. The environment mirrors a production Linux ops training setup.

Deploy K3s

Two VMs are used:

K3s node (2C4G) – Ubuntu 22.04, K3s v1.28.8, IP 10.16.255.1/16

VPN server (1C2G) – CentOS 7.9, LAN 10.16.255.254/16 as K3s gateway

Configure Shared Internet

Enable IP forwarding and NAT on the VPN server:

# echo 1 > /proc/sys/net/ipv4/ip_forward

# iptables -t nat -A POSTROUTING -s 10.16.0.0/16 -j MASQUERADE

Temporarily stop firewalld and SELinux:

# systemctl stop firewalld

# setenforce 0

Install K3s

Use the Rancher mirror and disable default network plugins:

export INSTALL_K3S_VERSION=v1.28.8+k3s1

export INSTALL_K3S_MIRROR=cn

wget https://rancher-mirror.rancher.cn/k3s/k3s-install.sh

sh k3s-install.sh --flannel-backend=none --disable-network-policy --disable=traefik --write-kubeconfig-mode 644 --write-kubeconfig ~/.kube/config --kubelet-arg=max-pods=5000

Deploy Kube‑OVN

Install the Kube‑OVN 1.12 CNI plugin:

wget http://stu.jxit.net.cn:88/qdcloud/kube-ovn-1.12-k3s-1.28.8-install.sh

bash -x kube-ovn-1.12-k3s-1.28.8-install.sh

Wait until all OVN pods are Running .

Deploy Multus

Install the Intel multus CNI and enable VPC NAT gateway support:

kubectl apply -f src/main/webapp/WEB-INF/cgi/yml/public/multus-daemonset-thick.yml

wget http://stu.jxit.net.cn:88/k8s/kube-ovn/multus-cni_4.0.2_linux_amd64.tar.gz

tar -zxf multus-cni_4.0.2_linux_amd64.tar.gz

cp multus-cni_4.0.2_linux_amd64/multus* /opt/cni/bin/

kubectl apply -f src/main/webapp/WEB-INF/cgi/yml/public/enable-vpc-nat-gw.yml

Modify external-network.yml to use the host NIC (e.g., ens33 ) and the correct CIDR/gateway.

Install cloud‑hypervisor

Download and make executable:

wget http://stu.jxit.net.cn:88/qdcloud/cloud-hypervisor -O /usr/bin/cloud-hypervisor

chmod +x /usr/bin/cloud-hypervisor

Deploy VPN Server

Install OpenVPN, copy configuration, and start it:

wget http://stu.jxit.net.cn:88/qdcloud/openvpn -O /usr/bin/openvpn

chmod +x /usr/bin/openvpn

nohup openvpn --config /etc/openvpn/openvpn.conf --client-config-dir /etc/openvpn/ccd --crl-verify /etc/openvpn/crl.pem &

Configure kubectl on VPN Server

Copy kubeconfig and kubectl binaries, then edit the server address to https://10.16.255.1:6443 and verify pod status.

Deploy Management Application (Tomcat + Maven)

Install Tomcat8, change listening port to 888, and build the project:

cd /opt/ && curl -O http://stu.jxit.net.cn:88/k8s/tomcat8-cgi.tar.gz && tar -zxf tomcat8-cgi.tar.gz && rm -f tomcat8-cgi.tar.gz

sed -i "s/port=\"80\"/port=\"888\"/g" /opt/tomcat8/conf/server.xml

Install required JARs and package the WAR:

mvn install:install-file -Dfile=lib/tangyuan-0.9.0.jar -DgroupId=org.xson -DartifactId=tangyuan -Dversion=0.9.0 -Dpackaging=jar

... (other jar installations) ...

mvn package -Dmaven.test.skip=true

cp target/qdcloud.war /opt/tomcat8/webapps/ROOT.war

/opt/tomcat8/bin/startup.sh

OpenVPN Client Setup

Download the Windows client, import the generated .ovpn file, adjust the server address, and connect to obtain an IP in the 10.32.0.0/24 range.

Accessing VMs and Containers

Use the management UI (port 888) to start/stop containers and VMs. SSH into containers (e.g., ssh [email protected] ) or VMs (e.g., ssh [email protected] ) after VPN connection.

Each VM gets a persistent PVC for its disk image, while containers use temporary storage.

Conclusion

Kube‑OVN provides secure, multi‑tenant networking for both containers and VMs, enabling a cloud‑native lab that integrates Kubernetes, virtual machines, VPN access, and custom applications.