Troubleshooting BFD Neighbor Failure on Cisco Nexus Switches with VPC Peer‑Gateway

1. Scenario Introduction

Two Cisco Nexus switches are connected via VPC to provide dual uplink access for servers. Each server uses a bonded interface (active‑standby) to connect to both switches. The server, acting as a Kubernetes node, establishes iBGP with both switches and uses single‑hop BFD for rapid link‑failure detection. Traffic from the server to SW2 must traverse the VPC peer‑link.

2. Problem Description

The BGP session between the server and SW2 is established, but the BFD neighbor cannot be formed.

3. Diagnosis

Packet captures show that SW2 receives the server’s BFD packets and replies with an ICMP “port unreachable” message, indicating that SW2 does not process the BFD packets.

Further analysis reveals that the BFD packets arriving at SW2 have a TTL of 254 (instead of the expected 255) and a source MAC address belonging to SW1. This shows that SW1 performed a Layer‑3 forward of the packets, decrementing the TTL and replacing the source MAC.

Cisco NX‑OS documentation states that, for security reasons, a switch will discard directly‑connected BFD packets whose TTL is not 255. Because SW2 sees a TTL of 254, it discards the packet and sends an ICMP error.

4. VPC Peer‑Gateway Explanation

When VPC peer‑gateway is enabled, a switch acts as a gateway for packets whose destination MAC is the MAC of its VPC peer. The switch copies its local interface vlan MAC to the peer, marking it with a G (gateway) flag in the peer’s MAC table. This reduces Layer‑2 traffic across the peer‑link by performing Layer‑3 forwarding for those MAC addresses.

5. Test Lab

Three test cases were performed on the same topology as described above.

Test 1: SW1 enables peer‑gateway, SW2 disables it

After enabling peer‑gateway on SW1, its interface vlan 400 MAC (00ea.bd5f.dfa7) appears in SW2’s MAC table with a G flag. Packets destined for this MAC are Layer‑3 forwarded by SW2, causing the source MAC to be replaced and the TTL to be decremented. SW2’s own interface vlan 400 MAC (6c8b.d3ca.ff67) has no G flag in SW1’s table, so packets to it are forwarded at Layer‑2 without TTL change. The HSRP virtual MAC (0000.0c07.ac01) has a G flag on both switches, so packets to the HSRP VIP are always Layer‑3 forwarded.

Test 2: SW1 disables peer‑gateway, SW2 enables it

Disabling peer‑gateway on SW1 removes the G flag for its VLAN MAC in SW2’s table. Enabling peer‑gateway on SW2 adds a G flag for its VLAN MAC in SW1’s table, while the HSRP VIP continues to have G flags on both sides.

Test 3: Both switches disable peer‑gateway

When both switches have peer‑gateway disabled, the G flags disappear from the VLAN MAC entries in each other’s tables. The HSRP VIP MAC still retains G flags, so packets to the HSRP VIP are still Layer‑3 forwarded locally.

6. Conclusions

Enabling VPC peer‑gateway adds a G flag to the local interface vlan MAC in the peer’s MAC table, causing the peer to perform Layer‑3 forwarding for packets destined to that MAC. Regardless of peer‑gateway status, the HSRP VIP MAC retains a G flag on both switches, ensuring local Layer‑3 forwarding for traffic to the VIP.

7. Resolution

The BFD neighbor failure was caused by VPC peer‑gateway causing SW1 to Layer‑3 forward the server’s BFD packets to SW2, reducing the TTL to 254. Disabling peer‑gateway stops the Layer‑3 forwarding, preserves the TTL at 255, and allows the BFD neighbor to be established successfully.

8. Notes

Testing also showed that a VPC peer‑link without LACP can affect BFD neighbor establishment across the link. It is recommended to configure the peer‑link as a port‑channel using LACP.

9. References

BFD RFC: https://datatracker.ietf.org/doc/html/rfc5880

Cisco NX‑OS BFD: Cisco NX‑OS BFD guide

Cisco VPC Peer‑gateway: https://community.cisco.com/t5/switching/vpc-peer-gateway/td-p/2726210

BFD over LACP port‑channel: https://www.cisco.com/c/en/us/support/docs/switches/nexus-7000-series-switches/200871-BFD-for-BGP-in-AdminDown-State-on-Nexus7.html