A Miswritten iptables Rule That Almost Made Me Quit

Incident Overview

On 2024‑Q3 a production server running CentOS 7, Java micro‑services and kernel 6.8.0 experienced a complete loss of SSH connectivity for 47 minutes. The root cause was a DROP rule that was inserted at the top of the INPUT chain: iptables -I INPUT -s 10.0.0.0/8 -j DROP The rule unintentionally matched the load‑balancer health‑check traffic (source 10.244.0.0/16), causing the health checks to fail, the backend nodes to be removed from the pool, and the frontend to return HTTP 502 errors.

Timeline (local time):

14:32:00 – Engineer executed the rule change.

14:32:15 – First health‑check failure.

14:34:47 – Five consecutive failures triggered backend removal.

14:35:12 – Massive 502 errors observed by users.

14:41:33 – Ops staff logged in via VNC (the only reachable console).

14:47:00 – Rule rollback completed, service restored.

14:52:00 – All backend nodes re‑joined the load‑balancer.

iptables Core Concepts

iptables is a user‑space front‑end to the Netfilter framework. Packets traverse five hook points – PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING – and are processed by tables ( filter, nat, mangle, raw). Rules are evaluated sequentially; the first matching rule decides the fate of the packet (first‑match semantics).

State tracking ( conntrack) provides the NEW, ESTABLISHED, RELATED and INVALID states. A typical minimal filter configuration looks like:

# Flush existing rules
iptables -F
iptables -X
# Default policies (least‑privilege)
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
# Allow loopback
iptables -A INPUT -i lo -j ACCEPT
# Allow established/related traffic
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allow SSH from management subnet
iptables -A INPUT -p tcp -s 10.0.0.0/24 --dport 22 -m state --state NEW -j ACCEPT
# Allow HTTP/HTTPS
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
# Log and drop everything else
iptables -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables-DROP: "
iptables -A INPUT -j DROP

Common Pitfalls

Rule order errors : A DROP placed before an ACCEPT will never be reached.

Interface confusion : Mixing INPUT, OUTPUT and FORWARD chains leads to rules being applied to the wrong traffic.

Port mismatches : Using UDP for a TCP service or specifying the wrong port number.

Overly broad source/destination : -s 0.0.0.0/0 or -d 0.0.0.0/0 opens the host.

Missing state rules : Without ESTABLISHED,RELATED the three‑way TCP handshake cannot complete.

Step‑by‑Step Troubleshooting

Emergency Response

If SSH is lost, use an out‑of‑band console (VNC, iLO, iDRAC) to gain access. Then:

# Backup current rules (recommended before any change)
iptables-save > /root/iptables-backup-$(date +%Y%m%d-%H%M%S).bak
# Flush all rules only after confirming you have console access
iptables -F
# Restore from backup if needed
iptables-restore < /root/iptables-backup-$(date +%Y%m%d-%H%M%S).bak

Diagnostic Commands

List rules with counters: iptables -L INPUT -n -v --line-numbers Show raw rules: iptables-save Inspect connection tracking: cat /proc/net/nf_conntrack | head -10 Test rule existence without applying:

iptables -C INPUT -p tcp --dport 22 -j ACCEPT

Typical Scenarios

SSH disconnect : Ensure the ESTABLISHED,RELATED rule is above any DROP and that a NEW rule for port 22 exists before a blanket DROP.

Web service unreachable : Verify tcp dport 80,443 rules exist and that the default policy on OUTPUT is ACCEPT (or add explicit outbound rules).

Container network failure : Check Docker’s automatically inserted DOCKER chain and make sure custom rules are not placed before Docker’s rules.

Production Best Practices

Least‑privilege default policies : Set INPUT and FORWARD to DROP, OUTPUT to ACCEPT.

Layered defense : Combine cloud security groups, host iptables, and application‑level ACLs.

Version‑controlled rule files : Store rules in /etc/sysconfig/iptables (RHEL/CentOS) or /etc/nftables.conf (Debian/Ubuntu) and keep them in Git.

Readable structure : Group rules by function (loopback, state, services, logging, final drop) and add comments.

Ansible Automation Example

# ansible-playbook: deploy firewall on web servers
- name: Backup existing iptables rules
  command: iptables-save > /root/iptables-backup-{{ ansible_date_time.iso8601 }}.save
  changed_when: false

- name: Ensure iptables package is installed
  package:
    name: iptables
    state: present

- name: Set default policies
  iptables:
    chain: INPUT
    policy: DROP
- iptables:
    chain: FORWARD
    policy: DROP
- iptables:
    chain: OUTPUT
    policy: ACCEPT

- name: Allow loopback
  iptables:
    chain: INPUT
    in_interface: lo
    jump: ACCEPT

- name: Allow established/related connections
  iptables:
    chain: INPUT
    ctstate: ESTABLISHED,RELATED
    jump: ACCEPT

- name: Allow SSH from management network
  iptables:
    chain: INPUT
    protocol: tcp
    dport: 22
    source: 10.0.0.0/24
    ctstate: NEW
    jump: ACCEPT

- name: Allow HTTP/HTTPS
  iptables:
    chain: INPUT
    protocol: tcp
    dport: "{{ item }}"
    jump: ACCEPT
  loop:
    - 80
    - 443

- name: Log and drop everything else
  iptables:
    chain: INPUT
    jump: LOG
    log_prefix: "iptables-DROP: "
- iptables:
    chain: INPUT
    jump: DROP

- name: Enable and start iptables service
  systemd:
    name: iptables
    state: started
    enabled: yes

Rollback Script

#!/bin/bash
BACKUP_DIR="/root/iptables-backups"
LATEST=$(ls -t ${BACKUP_DIR}/iptables-*.save | head -1)
if [[ -z "$LATEST" ]]; then
  echo "No backup found in $BACKUP_DIR"
  exit 1
fi
echo "Rolling back to $LATEST"
iptables-restore < "$LATEST"
if [[ $? -eq 0 ]]; then
  echo "Rollback successful"
  echo "iptables rollback performed on $(hostname)" | mail -s "ALERT: iptables rollback" [email protected]
else
  echo "Rollback failed"
  exit 2
fi

Monitoring & Alerting

Three lightweight scripts can be scheduled via cron (every 5 minutes) to detect configuration drift, abnormal drop rates, and conntrack saturation.

Configuration checksum monitor

# /usr/local/bin/check-iptables-checksum.sh
CHECKSUM_FILE="/var/lib/iptables/checksum"
CURRENT=$(md5sum /etc/sysconfig/iptables | awk '{print $1}')
if [[ -f "$CHECKSUM_FILE" ]]; then
  OLD=$(cat "$CHECKSUM_FILE")
  if [[ "$CURRENT" != "$OLD" ]]; then
    echo "WARNING: iptables configuration changed" | tee -a /var/log/iptables-monitor.log
    # Replace with your alerting command
    /usr/local/bin/send-alert.sh "iptables rules changed"
  fi
fi
echo "$CURRENT" > "$CHECKSUM_FILE"

Drop‑rate monitor

# /usr/local/bin/check-iptables-drops.sh
THRESHOLD=1000  # packets in the last 5 min
DROP_PKTS=$(iptables -L INPUT -n -v | awk '/ DROP /{sum+=$1} END{print sum}')
if (( DROP_PKTS > THRESHOLD )); then
  TS=$(date '+%Y-%m-%d %H:%M:%S')
  echo "$TS: WARNING – INPUT DROP packets = $DROP_PKTS" >> /var/log/iptables-drop.log
  /usr/local/bin/send-alert.sh "High DROP count: $DROP_PKTS"
fi

Conntrack saturation monitor

# /usr/local/bin/check-conntrack.sh
MAX=$(cat /proc/sys/net/netfilter/nf_conntrack_max)
CUR=$(cat /proc/net/nf_conntrack | wc -l)
PCT=$(( CUR * 100 / MAX ))
if (( PCT >= 90 )); then
  echo "CRITICAL: conntrack $PCT% full ($CUR/$MAX)" >> /var/log/conntrack.log
  /usr/local/bin/send-alert.sh "Conntrack table critical: $PCT%"
elif (( PCT >= 80 )); then
  echo "WARNING: conntrack $PCT% full ($CUR/$MAX)" >> /var/log/conntrack.log
fi

nftables Migration (2026 Recommendation)

nftables provides atomic rule updates, a unified table/chain model, and set collections. Migration can be performed in two ways:

Compatibility layer : Convert existing iptables rules with iptables-nftables-translate and test the output.

# Convert and test
iptables-save | iptables-nftables-translate > /tmp/nft.rules
nft -f /tmp/nft.rules -c   # syntax check only
# Apply if test passes
nft -f /tmp/nft.rules

Manual rewrite : Re‑implement the rule set using nftables syntax to take advantage of sets and maps.

# Minimal nftables filter table
nft add table ip filter
nft add chain ip filter INPUT { type filter hook input priority 0 \; policy drop }
# Loopback
nft add rule ip filter INPUT iif lo accept
# Established/related
nft add rule ip filter INPUT ct state established,related accept
# SSH from management subnet
nft add rule ip filter INPUT ip saddr 10.0.0.0/24 tcp dport 22 ct state new accept
# HTTP/HTTPS
nft add rule ip filter INPUT tcp dport {80,443} accept
# Log and drop
nft add rule ip filter INPUT limit rate 5/min log prefix "nft-DROP: " counter drop
# Save
nft list ruleset > /etc/nftables.conf
# Enable service
systemctl enable nftables && systemctl start nftables

When using the compatibility layer, verify that all iptables extensions have an nftables equivalent; some complex matches may require manual adjustment.

References

Netfilter/iptables Project – https://www.netfilter.org/projects/iptables/index.html

Red Hat Enterprise Linux 9 – Securing Networks – https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/securing_networks/index

Arch Wiki – iptables – https://wiki.archlinux.org/title/iptables

Docker Documentation – iptables integration – https://docs.docker.com/network/iptables/

Kubernetes – kube-proxy iptables mode – https://kubernetes.io/docs/reference/networking/virtual-ips/

Paul 丙戌 – 《iptables快速上手与实战》 (2025)

Red Hat – nftables release notes (2026) – https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/nftables_release_notes