How to Diagnose and Fix OpenStack Neutron Security Group Packet Drops

Preface

This article originates from the ADDOPS team. The author, Li Wenxin, is a cloud platform containerization and virtualization operations engineer at 360 HULK, responsible for network module design and development. It records a recent OpenStack Neutron security‑group problem and explains the troubleshooting process.

1 Determine Network Architecture

Understanding where packets are dropped requires familiarity with the full OpenStack Neutron data‑plane architecture. Assuming the physical hosts of the Linux Director and the virtual machines are reachable, the basic data‑plane diagram is shown below.

The diagram reveals that br‑ethx and br‑int are OVS bridges, while qbr‑XXX is a Linux bridge. The extra Linux bridge is needed because the VM’s tap device is attached to qbr‑XXX rather than directly to br‑int .

2 Packet Capture

Capture packets on both the physical host and the VM. The client request is observed at points A‑G; packets appear at points C‑G but not at A‑B, indicating a drop between the Linux bridge and the tap device.

tcpdump -i xxx -nv | grep ipip

3 Analyze the Problem

The issue lies between points B and C, i.e., between the Linux bridge and the tap device. Neutron uses iptables to enforce security‑group policies here, and the drop suggests a problem with those rules. Since SSH works, TCP traffic is fine, pointing to iptables misconfiguration.

4 Neutron Security‑Group iptables Rules

The Neutron L2 agent creates iptables chains for each VM tap device (i, o, s). The main functions are:

prepare, update, remove firewall filters

drop all packets by default

prevent IP spoofing based on MAC address

allow DHCP and ICMPv6 RA inbound

block outbound DHCP

drop INVALID packets

allow stateful, established connections

convert security‑group rules to iptables rules (IPv4, IPv6, TCP, UDP, ICMP, ICMPv6)

support multiple ports per rule via the multiport module

5 Chain Tracing

Inspect the iptables chains created by Neutron.

iptables -S | grep neutron

Key chains include neutron-filter-top , neutron-openvswi-FORWARD , neutron-openvswi-INPUT , neutron-openvswi-OUTPUT , and per‑VM chains such as neutron-openvswi-ibddxxxxc-b .

iptables -nxvL FORWARD

Shows the forward chain forwarding to neutron-filter-top and then to neutron-openvswi-FORWARD .

iptables -nxvL neutron-openvswi-FORWARD

Shows traffic being sent to the security‑group chain.

iptables -nxvL neutron-openvswi-sg-chain

Routes packets to VM‑specific chains ( neutron-openvswi-ibddxxxxc-b and neutron-openvswi-obddxxxxc-b ) or accepts them.

iptables -nxvL neutron-openvswi-ibddxxxxc-b

Handles inbound traffic for the VM; unmatched packets fall through to the fallback chain.

iptables -nxvL neutron-openvswi-sg-fallback

Default drop rule for unmatched traffic.

6 Neutron Security‑Group Whitelist Mechanism

Neutron security groups operate as a whitelist: only traffic matching user‑defined rules is allowed; everything else is dropped. The current rules do not permit the IPIP protocol, so IPIP packets are dropped.

7 Confirm IPIP Protocol Number

Kernel header include/uapi/linux/in.h defines IPIP as protocol number 4.

enum { IPPROTO_IP = 0, IPPROTO_ICMP = 1, IPPROTO_IGMP = 2, IPPROTO_IPIP = 4, ... }

8 Add Rule to Fix the Network

In the OpenStack Neutron security‑group dashboard, add a rule that allows IPIP (protocol 4). After applying, the control node generates the corresponding iptables rule.

Verification on the control node shows the new rule:

9 Unit Test

Capture packets on the VM again; the client receives replies and IPIP packets are observed. The iptables counters for protocol 4 are non‑zero, confirming the rule works.

iptables -nxvL neutron-openvswi-ibddxxxxc-b

The chain now shows a rule matching protocol 4 with packet counters.

10 Push to Production and Regression Test

The same steps are performed in the production environment and the results match the test environment, confirming the fix.

Conclusion

Through this exploration we learned how OpenStack Neutron security groups are implemented with iptables, gained insight into the Neutron data‑plane topology, and demonstrated a systematic method for tracing and fixing packet‑drop issues. Future articles will dive deeper into iptables pattern matching, advanced usage, and netfilter fundamentals.