Enterprise Firewall Configuration: Mastering iptables and nftables Rule Management

Overview

Netfilter is the Linux kernel firewall core. It registers five hook points— PREROUTING, INPUT, FORWARD, OUTPUT and POSTROUTING —that cover the full packet lifecycle. The historic iptables tool stores rules in a linear list; each packet traverses the list, so performance degrades sharply when the rule count reaches thousands.

Since kernel 3.13, nftables provides a unified framework that replaces iptables, ip6tables, arptables and ebtables. It introduces native sets and maps, hash‑based lookups, and atomic, transactional rule updates, delivering constant‑time matching and easier maintenance.

iptables vs nftables comparison

Kernel interface : iptables uses separate x_tables modules per protocol family; nftables uses a unified nf_tables subsystem.

User‑space tool : iptables/ip6tables/arptables/ebtables vs a single nft command.

Rule storage : linear list (O(n) per packet) vs hash/set structures (O(1) lookup).

Rule changes : full‑chain replacement, non‑atomic vs atomic, transactional updates.

Set support : requires external ipset vs native set / map with timeout.

Syntax style : command‑line flags ( -A -j) vs declarative BPF‑like syntax.

Performance (10 k rules) : linear latency increase vs constant‑time lookup.

Compatibility layer : none vs iptables‑nft provides a drop‑in compatible CLI.

Key technical features

Five Netfilter hook points ( PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING) cover the entire packet lifecycle.

Atomic rule updates in nftables prevent a window of inconsistency during changes.

Native set and map structures store IP/port collections and verdict mappings with O(1) lookup.

Connection tracking ( conntrack) provides stateful firewall capabilities; table overflow drops new connections.

Typical use cases

Server inbound protection (web, database).

Gateway NAT (SNAT/MASQUERADE, DNAT).

DMZ three‑zone isolation.

Container network policies (Docker/Kubernetes).

Basic DDoS mitigation (SYN flood, port‑scan detection).

Environment requirements

Linux kernel 6.x (full nftables features from 6.1; LTS 6.6 recommended).

nftables user‑space tool 1.1.x.

iptables‑nft 1.8.9+ for the compatibility layer.

Supported OS: Ubuntu 24.04, Debian 12, RHEL 9 (Debian 12 and RHEL 9 default to nftables).

conntrack‑tools 1.4.7+.

Detailed steps

1. iptables basics and enterprise rule set

#!/bin/bash
set -euo pipefail
# Flush existing rules
iptables -F
iptables -X
iptables -Z
# Default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
# Loopback
iptables -A INPUT -i lo -j ACCEPT
# Established/related
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# Drop invalid
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
# SSH brute‑force limit (6 attempts/min per IP)
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 6 --name SSH -j DROP
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
# Web ports
iptables -A INPUT -p tcp -m multiport --dports 80,443 -m conntrack --ctstate NEW -j ACCEPT
# ICMP rate limit (5/s)
iptables -A INPUT -p icmp --icmp-type echo-request -m limit --limit 5/s --limit-burst 10 -j ACCEPT
iptables -A INPUT -p icmp --icmp-type echo-request -j DROP
# Log dropped packets (rate‑limited)
iptables -A INPUT -m limit --limit 10/min -j LOG --log-prefix "[FW-DROP] " --log-level 4

2. NAT configuration (iptables)

# SNAT (static public IP)
iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j SNAT --to-source 203.0.113.10
# MASQUERADE (dynamic public IP)
iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE
# DNAT (public 80 → internal web)
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 10.0.0.100:8080
# Forward chain for DNAT
iptables -A FORWARD -i eth0 -o eth1 -p tcp --dport 8080 -d 10.0.0.100 -m conntrack --ctstate NEW -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

3. nftables syntax and core concepts

All rules are placed under a single inet family, handling IPv4 and IPv6 uniformly.

# List rules
nft list ruleset
# Add a rule
nft add rule inet filter input tcp dport 80 accept
# Delete a rule by handle
nft delete rule inet filter input handle 10

3.1 Enterprise rule set (nftables)

#!/usr/sbin/nft -f
flush ruleset

table inet filter {
    set blacklist { type ipv4_addr; flags timeout; timeout 10m; }
    set whitelist { type ipv4_addr; elements = { 10.0.0.1, 10.0.0.2 } }
    set tcp_allowed { type inet_service; elements = { 22, 80, 443, 8080 } }
    set port_ratelimit { type inet_service : verdict; elements = { 22 : jump ssh_limit, 80 : accept, 443 : accept } }

    chain ssh_limit {
        ct state new meter ssh_meter { ip saddr limit rate 6/minute burst 4 packets } accept
        add @blacklist { ip saddr }
        log prefix "[SSH-BRUTE] " level warn
        drop
    }

    chain syn_protect {
        tcp flags syn limit rate 500/second burst 200 packets accept
        log prefix "[SYN-FLOOD] " level warn
        drop
    }

    chain input {
        type filter hook input priority 0; policy drop;
        iif "lo" accept
        ip saddr @blacklist drop
        ip saddr @whitelist accept
        ct state established,related accept
        ct state invalid drop
        iifname $LAN_IF tcp dport 22 ct state new jump ssh_limit
        iifname $WAN_IF tcp dport @tcp_allowed ct state new accept
        ip protocol icmp icmp type echo-request limit rate 5/second accept
        ip6 nexthdr icmpv6 accept
        limit rate 10/minute log prefix "[NFT-INPUT-DROP] " level warn
    }

    chain forward {
        type filter hook forward priority 0; policy drop;
        ct state established,related accept
        ct state invalid drop
        iifname $WAN_IF oifname $DMZ_IF tcp dport @dmz_services ct state new accept
        iifname $DMZ_IF oifname $LAN_IF ip daddr $LAN_NET tcp dport @dmz_to_lan_ports ct state new accept
        iifname $LAN_IF oifname $WAN_IF accept
        iifname $LAN_IF oifname $DMZ_IF accept
        limit rate 10/minute log prefix "[NFT-FWD-DROP] " level warn
    }

    chain output { type filter hook output priority 0; policy accept; }
}

table inet nat {
    chain prerouting { type nat hook prerouting priority -100; policy accept; }
    chain postrouting { type nat hook postrouting priority 100; policy accept; }
}

4. Migration from iptables to nftables

Use iptables-translate for one‑to‑one conversion, then validate manually because modules such as recent have no direct nftables equivalent.

# Translate a single rule
iptables-translate -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
# Translate an entire rule set
iptables-save | iptables-restore-translate > /etc/nftables-migrated.conf

5. Best practices and pitfalls

Place high‑frequency rules (e.g., ESTABLISHED,RELATED) at the top of the chain to minimise per‑packet processing latency.

Replace long linear rule lists with set / map structures to reduce matching overhead.

Audit rule hit counters regularly; prune zero‑hit rules.

Configure nf_conntrack_max based on expected concurrent connections (recommended: nf_conntrack_max = concurrent * 1.5).

When editing rules remotely, schedule a fallback at job that restores a known‑good configuration after a few minutes to avoid lock‑out.

Docker manipulates the nat table directly; use the DOCKER-USER chain for container‑level firewall policies.

6. IPv6 dual‑stack

nftables inet family handles IPv4 and IPv6 with a single rule set, unlike iptables which requires separate iptables and ip6tables. IPv6 requires explicit allowance of ICMPv6 neighbor discovery.

# IPv6 NDP (must be allowed)
nft add rule inet firewall input icmpv6 type { nd-neighbor-solicit, nd-neighbor-advert, nd-router-solicit, nd-router-advert } accept
# DHCPv6 client
nft add rule inet firewall input udp sport 547 udp dport 546 accept

7. Container environment

Docker creates its own iptables rules in the nat table and the DOCKER chain, bypassing user‑defined INPUT/FORWARD chains. Use the DOCKER-USER chain to enforce container‑level policies, or disable Docker’s iptables management via /etc/docker/daemon.json with "iptables": false.

8. Conntrack tuning

Required entries ≈ concurrent connections × 1.5.

Memory consumption ≈ entries × 288 bytes (kernel 5.x).

Adjust net.netfilter.nf_conntrack_max (e.g., sysctl -w net.netfilter.nf_conntrack_max=150000 for ~100 k concurrent connections).

Adjust hash size (usually max/4) via /sys/module/nf_conntrack/parameters/hashsize.

TCP timeout parameters: nf_conntrack_tcp_timeout_established=86400 (1 day)

nf_conntrack_tcp_timeout_time_wait=30

nf_conntrack_tcp_timeout_close_wait=60

Persist settings in /etc/sysctl.d/99-conntrack.conf and apply with sysctl -p /etc/sysctl.d/99-conntrack.conf.

9. Monitoring and troubleshooting

View current rule set: nft list ruleset or iptables -L -v -n --line-numbers.

Add temporary logging rules to debug specific ports.

Trace packet flow with nft monitor trace (kernel‑level) or iptables -t raw -A PREROUTING -j TRACE (logs to kernel).

Check conntrack usage:

cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

If the table is full, increase nf_conntrack_max and adjust timeouts.

Identify performance bottlenecks by examining rule counters; consolidate high‑hit rules into sets.

10. Metrics collection (Prometheus)

Node exporter already collects nf_conntrack metrics. A custom textfile collector can expose additional firewall metrics.

# Example Prometheus collector (bash)
#!/bin/bash
set -euo pipefail
TEXTFILE_DIR="/var/lib/node_exporter/textfile_collector"
METRICS_FILE="$TEXTFILE_DIR/firewall.prom"

COUNT=$(cat /proc/sys/net/netfilter/nf_conntrack_count)
MAX=$(cat /proc/sys/net/netfilter/nf_conntrack_max)
RATIO=$(awk "BEGIN {printf \"%.4f\", $COUNT/$MAX}")

echo "# HELP fw_conntrack_usage_ratio conntrack table usage ratio"
echo "# TYPE fw_conntrack_usage_ratio gauge"
echo "fw_conntrack_usage_ratio $RATIO" > $METRICS_FILE
# Additional metrics (rule count, drops) can be added similarly

11. Backup and restore

# Backup nftables rules
nft list ruleset > /etc/nftables.conf.backup
# Backup iptables (IPv4 & IPv6)
iptables-save > /etc/iptables.rules.v4
ip6tables-save > /etc/iptables.rules.v6
# Restore
nft -f /etc/nftables.conf.backup
iptables-restore < /etc/iptables.rules.v4
ip6tables-restore < /etc/iptables.rules.v6

Conclusion

nftables provides a unified inet family, native sets/maps, and atomic updates, delivering constant‑time rule lookup and simpler management compared with legacy iptables. Proper conntrack sizing, rule ordering, and observability are essential to maintain high availability and security in production environments.