Deploying Cilium with BGP for Cloud‑Native Networking and Security at Ctrip

Cilium has become one of the most popular cloud‑native networking solutions in recent years, leveraging eBPF for flexible, high‑performance networking and L3‑L7 security policies. Ctrip began using Cilium in production in 2019, and this article outlines the deployment experience and related security explorations.

1. Network Evolution Overview From 2013 to 2018, Ctrip migrated from physical machines to virtual machines and then to containers, while still using the Neutron+OVS stack. As Kubernetes adoption grew, the Neutron+OVS model showed scalability limits, prompting a switch to a Cilium+BGP combination.

2. Cloud‑Native Network Practice The team customized Cilium deployment using Docker‑Compose + Salt instead of the default DaemonSet, employed BIRD as the BGP agent for advanced routing features, added StatefulSet/AdvancedStatefulSet IP support, and tailored monitoring and alerts.

2.1 BGP Peering Model The topology separates host‑internal networking (handled by Cilium and eBPF) from cross‑host networking (handled by BGP and kernel routing). Each node runs BIRD as a BGP speaker, announces a /24 or /25 PodCIDR to two data‑center peers, and receives no routes from the peers, resulting in a pure L3 routing network that avoids routing table bloat on hosts.

2.2 Typical Traffic Path (Pod → Service) When a pod accesses a Service, eBPF performs DNAT to a backend pod IP, the kernel routes the packet to the appropriate node, the data‑center network forwards it based on learned PodCIDR routes, and the destination node’s eBPF programs enforce inbound policies before delivering the packet to the target pod.

2.3 L4/L7 Ingress Solution To expose services outside the cluster, a four‑layer load balancer (L4LB) built on Cilium+BGP+ECMP provides VIPs that can be attached to externalIPs or LoadBalancer services, enabling external access and integration with Istio ingress‑gateway for L7 traffic.

3. Cloud‑Native Security Attempts Cilium offers two core capabilities: eBPF‑based high‑performance networking and L3‑L7 security policies via CiliumNetworkPolicy (CNP). A simple CNP example is shown below:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "clustermesh-ingress-l4-policy"
  description: "demo: allow only employee to access protected-db"
spec:
  endpointSelector:
    matchLabels:
      app: protected-db
  ingress:
  - toPorts:
    - ports:
      - port: "6379"
        protocol: TCP
    fromEndpoints:
      - matchLabels:
          app: employee

The policy restricts inbound TCP traffic on port 6379 to pods labeled app=protected-db and only allows sources with the label app=employee . Implementing security at scale introduces challenges such as multi‑cluster environments and hybrid infrastructures (bare‑metal, OpenStack, Neutron‑powered Kubernetes, and Cilium‑based clusters).

3.1 Multi‑Cluster and Hybrid Challenges Applications may span multiple clusters and platforms, making unified security enforcement difficult. Ctrip’s approach limits inbound security to containers already on the Cilium network while gradually migrating other workloads.

3.2 ClusterMesh Extension Cilium’s built‑in ClusterMesh provides multi‑cluster connectivity. To handle external endpoints (VMs, bare‑metal, Neutron pods), a custom component synchronizes their lifecycle information into Cilium as pseudo‑endpoints, allowing the same inbound security policies to apply uniformly.

The extended ClusterMesh gives every Cilium agent a consistent global view of all workloads, enabling L3‑L4 security policies across heterogeneous environments, with future plans to support L7 policies.

4. Summary The article summarizes Ctrip’s practical experiences with Cilium for cloud‑native networking and security, including BGP integration, traffic flow analysis, L4 load balancing, security policy design, and multi‑cluster orchestration using ClusterMesh.