Isolated VPC Egress Redesign: Unified EIP Gateway with Deterministic Port Blocks

Background

Isolated VPCs on the company cloud platform must satisfy three constraints: default deny (no implicit connectivity to other internal 10/11 segments), dedicated public IP identity (outbound source address must be an exclusive isolated public IP, not a shared platform egress), and auditable paths (every allowed access maps to a specific policy and ticket). Isolation is enforced at the security group layer using a two-layer model: BASE allow + VPC default deny. This yields three default policies: inbound (classiclink traffic allowed, controlled by security groups), outbound (local VPC CIDR allowed; 10/8 and 11/8 denied; infrastructure IPs like DNS whitelisted), and outbound internet (forwarding plane unrestricted, but source address SNATed to an isolated public IP for a second layer of identity isolation). Security groups were chosen over a separate forwarding plane for lowest implementation cost, VM-level granularity with priority semantics, and natural integration with existing ACL ticketing.

Current Situation and Problems

Existing Standard Path Matrix

Over years, isolated VPC cross-domain access has evolved into a standard path matrix covering eight directions:

Internet · Outbound: SNAT + isolated public IP via unified EIP gateway

Internet · Inbound: EIP + LVS Layer 4/7 load balancing

VPC ↔ VPC · Full mesh: Cloud Connect Network (CCN route-based)

VPC → VPC · Controlled unidirectional: VPC NAT (SNAT as target VPC NAT IP)

VPC → VPC · Point-to-point service: PrivateLink (endpoint → target VIP)

VPC → IDC · Controlled egress: VPC NAT (SNAT as 10-segment source IP) + security groups

IDC → VPC · Service exposure: 100 special segment VIP cascaded to FNAT mapping to 10-segment VIP

VPC ↔ IDC · On-demand bidirectional: Classiclink (source address judgment + route advertisement)

The outbound internet path supports all isolated VPCs (sandbox, hosted services) and is the highest-frequency, most complex link.

Two Structural Defects of Traditional SNAT Mode

In traditional (normal) SNAT mode, each SNAT gateway in a snat_rule binds a distinct public EIP and independently performs BGP advertisement. This creates two structural defects in isolated VPC scenarios:

Public IP waste: The same isolated CIDR landing on different SNAT gateways requires different public IPs to ensure symmetric return traffic. Typically, each gateway in the cluster consumes at least one public IP, making it impossible for businesses that can whitelist only a few IPs.

Inability to unify public egress: While the public ingress is unified at the EIP gateway, SNAT gateways must advertise routes via BGP to attract return traffic, resulting in two distinct egress points (SNAT and EIP), breaking path auditability.

Solution Selection

The trade-off was between letting SNAT gateways directly bind EIPs (preserving direct egress, each gateway advertising return paths) versus introducing a dedicated EIP gateway as a unified egress. The former avoids tunnel overhead but scatters public identity across gateways, requiring binding and billing policies at every forwarding point and making paths unauditable. The latter converges public identity to a single binding point, at the cost of control-plane port-block orchestration and data-plane VXLAN double encapsulation. The latter was chosen, effectively decoupling "EIP ownership" from the forwarding plane as an independent responsibility.

Solution Design

The standard outbound internet path rests on four core decisions:

Public identity ownership: EIP BGP advertisement and return-path forwarding converge at the EIP gateway. Motivation: the last hop before internet is fixed at the EIP binding point, making the path deterministic and auditable.

Multi-gateway coexistence: A single EIP lands on multiple SNAT gateways, each allocated a non-overlapping port block. Motivation: eliminates 4-tuple conflicts and supports horizontal gateway scaling.

Gateway interconnection: SNAT gateways connect to the EIP gateway via VXLAN tunnels (fixed VNI). Motivation: inner packets retain private semantics; identity translation is deferred entirely to the EIP gateway.

DPVS responsibility boundary: DPVS handles only session management and SNAT rewrite, unaware of public identity. Motivation: keeps forwarding plane simple; identity semantics converge in the control plane.

Port Block Allocation Algorithm (Deterministic NAT)

Total port space 1025–65500 (64,476 ports) is pre-partitioned into equal-length blocks based on max_snat_nodes. Each master gateway exclusively owns one block. The control plane persists the tuple (gateway IP, port_low, port_high) to the WanIPPortAlloc table and syncs it to the EIP gateway for return-path mapping. This design converts the "runtime port negotiation" problem into a "control-plane static planning" problem: any session's source port inevitably falls within its hosting gateway's block, so the EIP gateway only needs the (wan_ip, port range) pair to determine the return tunnel, without maintaining session state.

Three consistency constraints accompany this design:

max_snat_nodes prohibited from online changes — block boundaries are derived from configuration; runtime changes would cause allocated blocks to misalign and overlap with new boundaries.

Blocks only grow, never shrink — when a gateway goes offline its block is not reclaimed; the index slot is retained to keep allocation records stable.

Explicit upper bound — if gateway count exceeds max_snat_nodes, the error PORT_BLOCKS_EXHAUSTED is returned, forbidding runtime block reshuffling.

Overall Architecture and Implementation

End-to-End Orchestration View

From a business perspective, provisioning is fully declarative: create a snat_rule → create a wan_ip with mode=isolated; the platform automatically orchestrates the rest.

Control Plane: LBC Isolated Mode

The entry point is creating a wan_ip with mode: "isolated" (IDC mandatory, used by control plane to distinguish traditional vs. isolated paths). Orchestration flow:

Key steps:

Gateway discovery: Filter role=master SNAT gateways from etcd online Agent registrations ( GetMasterSnatAgents). If none available, return NO_MASTER_AGENT and roll back the wan_ip, guaranteeing no half-configured state.

Port mapping distribution: Call the new fipctl snat-port-mapping interface to push (floating_ip, port_range, snat_node_ip) tuples per gateway. fipctl writes to its own etcd; Agents asynchronously execute fipadm snatadd to complete return-path mapping on the EIP gateway.

Elastic scaling: New gateways request additional allocations via POST /lvs/v2.0/wanips/{id}/port-allocs from free blocks; block exhaustion explicitly returns PORT_BLOCKS_EXHAUSTED with no implicit expansion. fipctl addresses are routed by IDC (mapping from fipctl_urls idc → URL). Notably, if the address cannot be resolved at call time, the entire segment is skipped with only a warning log — fipctl unavailability does not block the main flow by design, but the trade-off is silent misconfiguration failures, which should be caught by existence checks during initialization.

The same three consistency constraints apply (reiterated for emphasis): max_snat_nodes immutable after first isolated wan_ip creation; blocks only added, never reclaimed; explicit exhaustion error when limit exceeded.

Data Plane: DPVS isolate_public

Mark Distribution Chain

The isolate_public mark originates from keepalived configuration, passes through the dpvs_dest_compat compatibility structure across processes, and finally lands on each worker lcore's dp_vs_dest for per-packet reading by the data plane. The EIP gateway tunnel endpoint is defined by global configuration nat_gw_ip / nat_gw_vni (v3+ supports per-destination override):

# keepalived.conf — RS granularity declaration
real_server 10.1.1.11 0 {
    vni   100011
    hip   10.16.0.166
    isolate_public          # this VM's public traffic uniformly steered to EIP gateway
}

# dpvs.conf — EIP gateway tunnel endpoint
vtep_ip_defs {
    nat_gw_ip  10.220.30.1
    nat_gw_vni 7000001
}

Outbound: Three-Stage Egress Decision

Outbound encapsulation resides in __dp_vs_out_xmit_snat4(). After SNAT source address rewrite, a priority-based three-stage judgment occurs. The three-stage design ensures zero change to existing paths : unmarked RSs retain their original path byte-for-byte, which is the prerequisite for safe "append-mode" rollout.

Return: De-marking and MAC Direction Swap

Return encapsulation resides in __dp_vs_xmit_snat4(). Packets return to the VM along the original request tunnel; the inner Ethernet header swaps direction (dmac/smac exchanged). Two invariants, hardened from production incidents, govern this path:

Invariant 1 · Return judgment de-marked: Return path relies solely on the connection's own tunnel attributes ( conn->vxlan_id, vtep_peer_addr), not on dest->isolate_public. RS marks may be rewritten by keepalived during connection lifetime, while the return tunnel is fixed at connection establishment — depending on the mark would cause existing connections to misroute after a mark change.

Invariant 2 · Inner MAC swapped by direction: Request inner frame is src=VM_MAC / dst=GW_MAC; return must swap them, otherwise the VM discards the frame. Implementation uses the parameter order of vxlan_encap() (dmac, smac); the return call site swaps the arguments to achieve the swap.

Key Pitfall: SNAT Direction Semantics Counter-Intuitive

In SNAT mode, connections are initiated by internal VMs. dp_vs_in() marks the VM→public direction as DPVS_CONN_DIR_OUTBOUND (corresponds to conn->packet_out_xmit), while the public→VM return direction is INBOUND (corresponds to conn->packet_xmit) — opposite to the intuitive "in-request/out-response" mental model. Version 1 mistakenly attached encapsulation to the FNAT outbound path ( __dp_vs_out_xmit_fnat4) based on function name intuition, but isolated public VM traffic belongs to SNAT service and never enters that path, causing a complete fallback.

Supporting Mechanism: sapool Port Ranges

The data plane cooperates with control-plane port block planning: inet_ifaddr gains sapool_low / sapool_high passed through to sa_pool_create(), restricting source port allocation to the designated range. The Agent-side provisioning command evolves to:

dpip addr add {ip}/32 dev dpdk0 sapool low {port_low} high {port_high}

The low/high parameters correspond one-to-one with the port blocks recorded in control-plane WanIPPortAlloc. Across multiple gateway instances, source port ranges are mutually exclusive, eliminating 4-tuple conflicts at the EIP gateway mechanically. In isolated mode, the Agent also skips the BGP advertisement process (wan_ip reachability is handled by the EIP gateway).

Fault Diagnosis

The orchestration chain defines fault domain boundaries; troubleshooting proceeds by segment:

EIP creation fails and rolls back → Control plane gateway discovery → Typical cause: no master SNAT Agent in region ( NO_MASTER_AGENT).

Creation succeeds but public unreachable → Control plane → fipctl → Typical cause: missing IDC entry in fipctl_urls, EIP/port mapping silently skipped (warn log only); most frequent in legacy config upgrades.

Single gateway unreachable → Single node data plane → Typical cause: that gateway's WanIPPortAlloc.status=error; dpip sapool range inconsistent with port block.

Outbound works but return packets fail → DPVS return encapsulation → Typical cause: connection tunnel attributes lost; inner MAC not swapped by direction.

Worth emphasizing: fipctl_urls misconfiguration leads to silent failures — control plane calls are skipped without error, surfacing only at the far end of the chain, dramatically increasing debugging cost. Therefore, cross-system dependency configs should have existence checks at initialization, not rely on runtime logs.

Future Plans

Add initialization-time existence checks for cross-system configs like fipctl_urls, converting silent failures to fail-fast.

Include backup SNAT nodes in port block allocation and automatic failover.

Support dynamic port block shrink and reclaim, closing the loop with gateway decommissioning.

Agent health checks and automatic unhealthy node removal to reduce manual intervention.