How to Switch Calico Between IPIP, BGP, and Cross‑Subnet Modes in Kubernetes

Scope

The document applies to the following software versions:

Software

Version

Kubernetes

v1.14.x, v1.15.x, v1.16.x

Calico

v3.13.4

Overview

Calico is an open‑source networking and network‑security solution for containers, virtual machines, and host‑native workloads. It supports many platforms, including Kubernetes, Docker, OpenStack, and bare‑metal services, and offers multiple network modes.

BGP mode: Nodes act as virtual routers using the BGP protocol to provide inter‑pod connectivity.

IPIP mode: A new IP packet is encapsulated inside the original, rewriting source and destination IPs to the host IPs.

Cross‑subnet: Combines BGP for same‑subnet nodes and IPIP for cross‑subnet nodes.

Switch Calico to BGP Mode

After deployment, Calico defaults to IPIP mode; routes to other nodes go through the tunl0 interface.

To enable BGP mode, edit the calico-node DaemonSet in the system namespace.

Set CALICO_IPV4POOL_IPIP to off and add the environment variable FELIX_IPINIPENABLED=false.

Restart the nodes and verify that routes now use the physical network interface instead of tunl0.

Switch Calico to Cross‑Subnet Mode

Cross‑subnet mode is useful when some hosts span subnets and cannot use BGP. Same‑subnet machines use BGP, while cross‑subnet machines use IPIP.

After deploying Calico, the default IPIP mode creates a tunl0 tunnel interface on the host.

Routes to other hosts go through tunl0.

To switch, edit the IPPool: kubectl edit ipPool/default-ipv4-ippool Change ipipMode to crossSubnet.

Delete and recreate the calico-node pod via the UI, then verify the network.

Same‑subnet hosts use BGP, cross‑subnet hosts use IPIP via tunl0. Test connectivity with ping across hosts.

Configure Route Reflector

Install calicoctl

Installation methods:

Binary on a single host

Container on a single host

Run as a Kubernetes pod

Typical usage: install the binary on a worker node, ensure bird / felix are running, and use calicoctl for node‑specific commands.

curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.13.3/calicoctl

Make it executable and copy to /usr/bin:

chmod +x calicoctl

cp calicoctl /usr/bin/

Configure it to talk to the Kubernetes datastore:

export CALICO_DATASTORE_TYPE=kubernetes
export CALICO_KUBECONFIG=~/.kube/config
calicoctl node status

Calico node‑to‑node mesh

By default Calico uses node‑to‑node mesh. In large clusters, full mesh BGP peers consume resources, so a route reflector reduces the number of peer relationships.

Route reflector roles

Route reflector can be:

A node inside the cluster

A node outside the cluster running Calico

Other software or devices that support route reflection

Disable node‑to‑node mesh

cat <<EOF | calicoctl apply -f -
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  asNumber: 63400
EOF

Set Route Reflector

Configure a node as a route reflector by setting a cluster ID and labeling the node:

calicoctl patch node rke-node4 -p '{"spec": {"bgp": {"routeReflectorClusterID": "244.0.0.1"}}}'

kubectl label node rke-node4 route-reflector=true

Create a BGPPeer that selects the reflector:

export CALICO_DATASTORE_TYPE=kubernetes
export CALICO_KUBECONFIG=~/.kube/config
cat <<EOF | calicoctl apply -f -
kind: BGPPeer
apiVersion: projectcalico.org/v3
metadata:
  name: peer-with-route-reflectors
spec:
  nodeSelector: all()
  peerSelector: route-reflector == 'true'
EOF

Verify BGP status

After applying, the node’s peer type changes from mesh to node‑specific, and the route reflector shows a healthy connection.

Set veth Interface MTU

Use the highest MTU that does not cause fragmentation. For jumbo‑frame capable networks, configure Calico to use the appropriate MTU. IPIP adds a 20‑byte header; VXLAN adds a 50‑byte header, so subtract these from the physical MTU.

Set both workload endpoint MTU and tunnel MTU to the same value after upgrading the cluster.

Set Global AS Number

By default all Calico nodes use AS 64512. Change it by applying a new BGPConfiguration:

cat <<EOF | calicoctl apply -f -
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  asNumber: 64513
EOF

Set Per‑Node AS Number

Example: set node node-1 to AS 64514.

calicoctl patch node node-1 -p '{"spec": {"bgp": {"asNumber": "64514"}}}'

Modify Node Address Range

Calico allocates a default CIDR 10.42.0.0/16 with a 26‑bit subnet mask, giving each node up to 64 pods. Adjust block size to change address density and routing table size.

Note: Block size affects pod IP allocation and routing entry count; larger blocks reduce routes but also reduce node capacity.

To replace an IP pool:

Add a new IP pool (must be within the same cluster CIDR).

Disable the old pool (prevents new allocations but does not affect existing pods).

Delete pods from the old pool.

Verify new pods receive addresses from the new pool.

Delete the old pool.

Define a new IPPool:

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: my-ippool
spec:
  blockSize: 24
  cidr: 192.0.0.0/16
  ipipMode: Always
  natOutgoing: true

calicoctl apply -f pool.yaml

calicoctl patch ippool default-ipv4-ippool -p '{"spec": {"disabled": true}}'

Define IPPool by Node Labels

Label nodes to assign specific IP pools:

kubectl label nodes kube-node-0 rack=0
kubectl label nodes kube-node-1 rack=1

Create IPPools with node selectors:

calicoctl create -f - <<EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: rack-0-ippool
spec:
  cidr: 192.168.0.0/24
  ipipMode: Always
  natOutgoing: true
  nodeSelector: rack == "0"
EOF

calicoctl create -f - <<EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: rack-1-ippool
spec:
  cidr: 192.168.1.0/24
  ipipMode: Always
  natOutgoing: true
  nodeSelector: rack == "1"
EOF

Disable SNAT

By default Calico SNATs outbound traffic to the host IP. To apply firewall rules directly to pod IPs, set natOutgoing to false: kubectl edit ippool/default-ipv4-ippool Change natOutgoing: true to natOutgoing: false. After this, outbound traffic retains the pod IP.

Assign Fixed Pod IP

Single fixed IP via pod annotation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-test
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        "cni.projectcalico.org/ipAddrs": "[\"10.42.210.135\"]"
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

Multiple fixed IPs require an IPPool with natOutgoing: true and pod annotation cni.projectcalico.org/ipv4pools:

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: pool-1
spec:
  blockSize: 31
  cidr: 10.21.0.0/31
  ipipMode: Never
  natOutgoing: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-test
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        "cni.projectcalico.org/ipv4pools": "[\"pool-1\"]"
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80