How Hybridnet Enables On‑Anywhere Kubernetes Networking with Overlay & Underlay

Background

Hybridnet is the container network implementation used by Alibaba Cloud's CNStack platform. It is an open‑source, generic Kubernetes CNI plugin that can be deployed in any environment, similar to Flannel or Calico.

Repository:

https://github.com/alibaba/hybridnet

Design Goals

Simple architecture & stable functionality – essential for on‑premises delivery where troubleshooting cost is high.

“On Anywhere” deployment – the network must work without special prerequisites.

Flexibility – support diverse customer requirements and extensible use cases.

Easy operations – enable rapid onboarding for delivery engineers.

Key Concepts

Network (Network Domain)

Hybridnet introduces a Network CRD that groups nodes sharing the same network characteristics (e.g., same VLAN, gateway, or L2 segment). Pods belonging to a network domain can be scheduled onto any node in that domain.

apiVersion: networking.alibaba.com/v1
kind: Network
metadata:
  name: init
spec:
  netID: 4
  type: Overlay
status:
  dualStackStatistics:
    available: 65526
    total: 0
    used: 0
  ipv6Statistics:
    available: 65534
    total: 65535
    used: 1
  nodeList:
    - izf8z4x1svumpy31kcrltaz
    - izf8zatm3b8zaaks5gqk0tz

Subnet

A Subnet CRD is attached to a Network and defines a usable IP range for pods.

apiVersion: networking.alibaba.com/v1
kind: Subnet
metadata:
  name: init
spec:
  config:
    autoNatOutgoing: true
  network: init
  range:
    cidr: 172.24.0.0/16
    version: "4"
status:
  available: 65526
  total: 65534
  used: 8

Every Hybridnet cluster must contain at least one Network and one Subnet; together they form the user‑visible configuration entry point.

Policy‑Routing Data Plane

Hybridnet builds its data plane entirely on Linux policy routing, avoiding iptables/ipvs rules used by many other CNI solutions. This design is compatible with kube‑proxy, felix, and standard NetworkPolicy implementations.

Key routing tables added by Hybridnet:

Table 39999 – per‑pod routes to the pod’s veth device (highest priority).

Table 40000 – routes for the overlay CIDR via the VXLAN device.

Table 40001 – a default route that forces reverse traffic from other nodes to traverse the overlay tunnel.

Table 10000 – per‑network‑domain rules that match source IPs of a specific subnet and forward traffic according to the underlying mode (overlay, VLAN, or BGP).

Example of the policy‑routing rules on a node:

0: from all lookup local
1: from all lookup 39999
2: from all lookup 40000
3: from all fwmark 0x20/0x20 lookup 40001
4: from 172.24.0.0/16 fwmark 0x0/0x4040 lookup 10000
32766: from all lookup main
32767: from all lookup default

Overlay & Underlay Coexistence

Hybridnet supports both overlay (VXLAN) and underlay (VLAN or BGP) modes simultaneously. The overlay provides an “On Anywhere” abstraction, while underlay traffic is forwarded directly by the host NIC after customer‑provided configuration.

VXLAN (Overlay) Implementation

The VXLAN device behaves like a physical NIC; its FDB table maps destination MACs to VTEP IPs.

Each node configures proxy‑ARP/ND entries for all pod IPs on the VXLAN device, allowing the host to answer neighbor requests.

Static FDB entries with a zero MAC are created for each VTEP IP, forming a broadcast group that ensures packets are tunneled to all VTEPs when no specific entry matches.

A user‑space process listens for L2‑miss events, queries the IPInstance CRD, and returns the correct neighbor information, avoiding kernel broadcast storms.

VLAN (Underlay) Implementation

For VLAN mode, Hybridnet creates a virtual VLAN device for each non‑zero netID. Pods receive IPs from the VLAN subnet, and the host configures proxy‑ARP entries for those IPs. To prevent illegal ARP requests (where the host would use its own IP as the sender), Hybridnet introduces an “enhanced address” with noprefixroute, scope link, and no local route entry, ensuring a valid source address without consuming extra IP space.

BGP (Underlay) Implementation

In BGP mode, all pod traffic is routed through an external BGP gateway. Detailed design is documented in the Hybridnet wiki.

Hybridnet in CNStack

CNStack initializes an overlay network by default and treats it as the primary network for platform pods, guaranteeing management‑plane functionality in any environment. Users can add underlay networks via the CNStack console and annotate pods to use a specific underlay subnet, decoupling network planning from delivery.

References

https://github.com/alibaba/hybridnet

https://github.com/alibaba/hybridnet/wiki

https://github.com/alibaba/hybridnet/wiki/VXLAN-%E7%BD%91%E7%BB%9C

https://github.com/alibaba/hybridnet/wiki/VLAN-%E7%BD%91%E7%BB%9C

https://github.com/alibaba/hybridnet/wiki/BGP-%E7%BD%91%E7%BB%9C