Build a Multi‑Subnet Kubernetes Cluster with Calico BGP Using ENSP Simulation

Architecture Overview

The test builds a "per‑rack autonomous system" architecture, following the AS‑per‑Rack model. The environment consists of an ENSP‑based network (R1 and R2 routers) and a workstation‑hosted Kubernetes cluster spanning multiple subnets.

The Kubernetes cluster includes one master (10.30.81.127) and two nodes (10.30.81.128, 10.30.81.130) initially, later expanded with two additional nodes in the 192.168.219.0/24 subnet.

k8s‑master: 10.30.81.127

k8s‑node1: 10.30.81.128

k8s‑node2: 10.30.81.130

ENSP Configuration

Routers R1 and R2 are connected as eBGP peers. Static routes forward traffic between the 192.168.219.0/24 subnet and the internal 10.30.81.0/24 and 30.30.30.0/24 networks. BGP is configured on both routers, including iBGP peers for the same AS and eBGP peers toward the Kubernetes subnets.

# R1 configuration
int g0/0/0
ip a 20.20.20.10 24
int g0/0/1
ip a 10.30.81.118 24
int g0/0/2
ip a 30.30.30.10 24

# static routes
ip route-static 192.168.219.0 24 20.20.20.20
ip route-static 192.168.219.0 24 30.30.30.20

# BGP 64512
bgp 64512
peer 10.30.81.127 as-number 64512
peer 10.30.81.128 as-number 64512
peer 10.30.81.130 as-number 64512
peer 192.168.219.10 as-number 64513 ebgp-max-hop 5 connect-interface g0/0/1
peer 10.30.81.127 reflect-client
peer 10.30.81.128 reflect-client
peer 10.30.81.130 reflect-client

# R2 configuration
int g0/0/0
ip a 192.168.219.10 24
int g0/0/1
ip a 20.20.20.20 24
int g0/0/2
ip a 30.30.30.20 24

# static routes
ip route-static 10.30.81.0 24 20.20.20.10
ip route-static 10.30.81.0 24 30.30.30.10

# BGP 64513
bgp 64513
peer 10.30.81.118 as-number 64512 ebgp-max-hop 5 connect-interface g0/0/0
peer 192.168.219.20 as-number 64513
peer 192.168.219.40 as-number 64513
peer 192.168.219.20 reflect-client
peer 192.168.219.40 reflect-client

After configuring R1 and R2, pinging 10.30.81.118 from R2 succeeds, confirming the eBGP session. However, pinging any Kubernetes node from R2 fails because the nodes lack return routes. Adding the following routes on each node resolves the issue:

# Add routes on the three Kubernetes hosts
route add -n 20.20.20.0/24 gw 10.30.81.118 dev ens33
route add -n 30.30.30.0/24 gw 10.30.81.118 dev ens33

Adding New Machines to the Cluster

New VMs are created in NAT mode with static IPs in the 192.168.219.0/24 network. Example network configuration:

TYPE=Ethernet
DNS1=8.8.8.8
IPADDR=192.168.219.40
NETMASK=255.255.255.0
GATEWAY=192.168.219.10
BOOTPROTO=static
ONBOOT=yes

System preparation on each new node includes disabling firewalld, disabling SELinux, enabling IP forwarding and bridge netfilter, turning off swap, and applying sysctl settings:

systemctl stop firewalld && systemctl disable firewalld
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0
echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
echo 'net.bridge.bridge-nf-call-iptables = 1' >> /etc/sysctl.conf
swapoff -a
sysctl -p

Kubelet systemd unit is placed in /usr/lib/systemd/system/kubelet.service (or copied to /usr/bin) with --fail-swap-on=false. Docker is installed, and Calico and kube‑proxy images are imported from the existing cluster. Finally, kubeadm join adds the node to the cluster.

Verification of Routing

Traceroute from the master to a node in the 192.168.219.0/24 subnet shows the path passing through R2 (20.20.20.20), confirming that traffic traverses the ENSP network.

# traceroute 192.168.219.20
1  10.30.81.118  18ms
2  20.20.20.20  80ms
3  k8s-node3   110ms

Calico BGP Configuration

Calico is switched to pure BGP mode by setting nodeToNodeMeshEnabled: false in the BGPConfiguration. The IPIP and VXLAN pools are set to Never. Each node’s BGP AS number is set (64512 for the original rack, 64513 for the new rack) via calicoctl get node -o yaml and edited.

spec:
  bgp:
    asNumber: 64513

Nodes are labeled to group them by AS:

# AS 64512
kubectl label nodes k8s-master as-group=as64512 as-id=as64512
kubectl label nodes k8s-node1 as-group=as64512 as-id=as64512
kubectl label nodes k8s-node2 as-group=as64512 as-id=as64512

# AS 64513
kubectl label nodes k8s-node3 as-group=as64513 as-id=as64513
kubectl label nodes k8s-node4 as-group=as64513 as-id=as64513

BGPPeer objects are created so that nodes in each AS peer with the corresponding router (R1 or R2) while nodes sharing the same AS act as route‑reflector clients:

# as64513 nodes peer with R2
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: as64513-to-r2-peer
spec:
  nodeSelector: rr-id == 'as64513'
  peerIP: 192.168.219.10
  asNumber: 64513

# as64512 nodes peer with R1
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: as64512-to-r1-peer
spec:
  nodeSelector: rr-id == 'as64512'
  peerIP: 10.30.81.118
  asNumber: 64512

Apply the peers with calicoctl apply -f <file>.yaml.

Validating BGP Advertisement and Pod Connectivity

Pods are created on each node using a simple manifest that pins them to specific nodes via nodeSelector. After deployment, kubectl get pod -o wide shows the pod IPs on different subnets.

# Sample pod manifest (master.yaml)
apiVersion: v1
kind: Pod
metadata:
  name: master
spec:
  containers:
  - name: master
    image: larioy/nettool:latest
  nodeSelector:
    node: master

From the master pod, pinging a pod on node3 (different subnet) succeeds, confirming end‑to‑end BGP routing. Bird client output on the master node shows learned routes for all pod CIDRs.

R1 also learns the pod CIDR routes, as shown in the ENSP router screenshot.

Packet capture on R2 confirms that traffic between pods in different subnets traverses the BGP‑enabled path.

Manual BGP Maintenance Considerations

The ENSP BGP configuration is currently manual. For a per‑rack AS deployment, an automated solution is needed to detect host up/down events within an AS and update route‑reflector client information accordingly.

Architectural Reflections

The tested "per‑rack independent AS" design uses a top‑level switch layer where each rack’s route‑reflector establishes eBGP sessions with others. An alternative is a router‑centric top layer where all routers peer with every route‑reflector, eliminating inter‑RR BGP sessions. Future work should evaluate scalability as route entries grow.

Issue Log

ENSP could not find the vmnet8 interface; reinstalling WinPcap resolved it.

ENSP router devices sometimes become invalid; clearing and re‑adding devices helped.

Workstation NAT mode prevented VM connectivity; ensuring the NAT range matches the vmnet8 subnet fixed it.

When adding new nodes, switch back to bridged networking to install tools, then return to NAT before joining the cluster.

For ENSP network troubleshooting, capture packets at both ends of R1 and R2 interfaces.

Original article: https://larioy.gst.monster/2021/09/05/k8s-ji-chong-cni-fang-an-jie-xi/calico/ensp-mo-ni-calico-kua-wang-duan-bgp-wang-luo/