Deploy Flannel CNI with Host‑GW Model and Optimize iptables in a Kubernetes Cluster

1. Introduction

1.1 Preface

Kubernetes defines a network model but delegates its implementation to CNI plugins. Common CNI plugins include Flannel, Calico, Canal, Contiv, OpenContrail, NSX‑T, and Kube‑router.

1.2 Three Flannel Network Models

host‑gw model: all nodes must share the same physical gateway; it works by adding a static route on the host.

Vxlan model: used when hosts are on different subnets; it creates a virtual network device on each host and communicates via a virtual tunnel.

direct routing model: uses host‑gw when nodes share a gateway and Vxlan otherwise.

2. Cluster Planning

Example deployment uses 10.4.7.21 as the first node; 10.4.7.22 follows the same steps.

主机名   角色   IP
hdss7-21  Flannel 10.4.7.21
hdss7-22  Flannel 10.4.7.22

3. Download Software, Extract, and Create Symlink

Download address: https://github.com/flannel-io/flannel/

# cd /opt/src/
# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
# mkdir /opt/flannel-v0.11.0
# tar -zxvf flannel-v0.11.0-linux-amd64.tar.gz -C /opt/flannel-v0.11.0
# ln -s /opt/flannel-v0.11.0 /opt/flannel

4. Final Directory Structure

# cd /opt/flannel
# ls
flanneld  mk-docker-opts.sh  README.md

5. Copy Client Certificates

Used as etcd client.

# mkdir cert
# cd cert/
# scp hdss7-200:/opt/certs/ca.pem .
# scp hdss7-200:/opt/certs/client.pem .
# scp hdss7-200:/opt/certs/client-key.pem .
# ll

6. Create Configuration

Note: each host in the Flannel cluster has different configuration; on 10.4.7.22 set FLANNEL_SUBNET=172.7.22.1/24.

FLANNEL_NETWORK=172.7.0.0/16
FLANNEL_SUBNET=172.7.21.1/24
FLANNEL_MTU=1500
FLANNEL_IPMASQ=false

7. Create Startup Script

Adjust --public-ip and --iface as needed.

# vim flanneld.sh
#!/bin/sh
./flanneld \
  --public-ip=10.4.7.21 \
  --etcd-endpoints=https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
  --etcd-keyfile=./cert/client-key.pem \
  --etcd-certfile=./cert/client.pem \
  --etcd-cafile=./cert/ca.pem \
  --iface=ens33 \
  --subnet-file=./subnet.env \
  --healthz-port=2401

8. Check Permissions and Create Log Directory

# chmod +x flanneld.sh
# mkdir -p /data/logs/flanneld

9. Operate etcd to Add host‑gw

Run on any of the etcd nodes (12, 21, 22).

# cd /opt/etcd
# ./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}'
# ./etcdctl get /coreos.com/network/config

10. Create Supervisor Configuration

# vim /etc/supervisord.d/flannel.ini
[program:flanneld-7-21]
command=/opt/flannel/flanneld.sh
numprocs=1
directory=/opt/flannel
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/flanneld/flanneld.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=4
stdout_capture_maxbytes=1MB
stdout_events_enabled=false

11. Start Service and Verify

# supervisorctl update
# tail -100f /data/logs/flanneld/flanneld.stdout.log

12. Deploy Other Nodes and Verify Services

Repeat steps 3‑11 on 10.4.7.22. After Flannel starts, routing tables show that traffic to the other node’s container subnet goes through the host, confirming Flannel’s operation (host‑gw requires all hosts under the same gateway).

# route

13. Verify Pod Network Connectivity

From node 21, access a container on node 22; communication succeeds.

# curl 172.7.22.2

14. Optimize iptables Rules on Compute Nodes

14.1 Add curl to Container Image

Replace the image with a curl‑enabled version.

# vim /opt/kubernetes/conf/nginx-ds.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ds
spec:
  template:
    metadata:
      labels:
        app: nginx-ds
    spec:
      containers:
      - name: my-nginx
        image: harbor.od.com/public/nginx:curl
        ports:
        - containerPort: 80
# kubectl apply -f /opt/kubernetes/conf/nginx-ds.yaml

14.2 Pre‑Optimization Access

Before changing iptables, container‑to‑container traffic appears to originate from the host IP due to SNAT.

# kubectl exec -it nginx-ds-nqb57 bash
# curl 172.7.22.2

14.3 Start Optimizing iptables

On node 21, install iptables‑services and adjust NAT rules to exclude intra‑cluster traffic.

# yum install iptables-services -y
# systemctl start iptables && systemctl enable iptables
# iptables -t nat -D POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
# iptables -t nat -I POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
# iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited
# iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited

15. Save iptables Rules on Compute Nodes

15.1 Save Rules

# iptables-save > /etc/sysconfig/iptables
# service iptables save

15.2 Restart Docker Service

Restart Docker to reapply NAT rules; filter rules remain unchanged.

# systemctl restart docker
# iptables-save | grep -i postrouting | grep docker0

16. Troubleshooting

If a pod cannot reach the external network, compare the DNS settings of a Docker‑run container (which works) with the pod’s /etc/resolv.conf. Adjust the pod’s nameserver to a reachable DNS (e.g., 10.4.7.11) and reinstall needed tools.

# echo "nameserver 10.4.7.11" > /etc/resolv.conf
# ping baidu.com
# apt-get update && apt-get install curl -y
# curl -k https://www.baidu.com