Understanding Linux Netfilter: How the Kernel Handles Packet Filtering

What Is Netfilter?

Netfilter, introduced in Linux 2.4, is a kernel‑level framework that offers a set of hook functions for packet filtering, network address translation (NAT), and connection tracking. It sits between the network device driver and the protocol stack, allowing user‑space tools like iptables to register callbacks at predefined points.

The packet flow can be visualized as a "stack" where each layer adds a header on transmission and strips it on reception, eventually delivering raw data to the application.

Key Hook Points and Return Values

Netfilter defines five critical hook points (A‑E) in the IPv4 stack, renamed in netfilter_ipv4.h as PRE_ROUTING, LOCAL_IN, FORWARD, LOCAL_OUT, and POST_ROUTING. At each point, registered hook functions are invoked in order of priority and must return one of the following verdicts:

NF_ACCEPT – continue normal packet processing.

NF_DROP – discard the packet.

NF_STOLEN – the hook takes ownership of the packet; Netfilter does not free it.

NF_QUEUE – queue the packet to user‑space for further handling.

NF_REPEAT – re‑invoke the hook (use with caution to avoid loops).

Core Macros: NF_HOOK and NF_HOOK_THRESH

The macro NF_HOOK(pf, hook, skb, indev, outdev, okfn) inserts a packet into the Netfilter framework. In Linux 2.6, a more flexible version NF_HOOK_THRESH adds a thresh parameter to limit traversal to hooks with priority greater than the threshold.

#define NF_HOOK(pf, hook, skb, indev, outdev, okfn) \
    NF_HOOK_THRESH(pf, hook, skb, indev, outdev, okfn, INT_MIN)

#define NF_HOOK_THRESH(pf, hook, skb, indev, outdev, okfn, thresh) \
({ int __ret; \
   if ((__ret = nf_hook_thresh(pf, hook, &skb, indev, outdev, okfn, thresh, 1)) == 1) \
       __ret = (okfn)(skb); \
   __ret; })

The helper function nf_hook_thresh checks a condition flag and either skips processing or calls nf_hook_slow, which iterates over the registered hook list respecting priority.

static inline int nf_hook_thresh(int pf, unsigned int hook,
                                 struct sk_buff **pskb,
                                 struct net_device *indev,
                                 struct net_device *outdev,
                                 int (*okfn)(struct sk_buff *), int thresh,
                                 int cond)
{
    if (!cond)
        return 1;
    #ifndef CONFIG_NETFILTER_DEBUG
    if (list_empty(&nf_hooks[pf][hook]))
        return 1;
    #endif
    return nf_hook_slow(pf, hook, pskb, indev, outdev, okfn, thresh);
}

Hook Points in the IPv4 Stack

The kernel stores callbacks in a two‑dimensional array struct list_head nf_hooks[NPROTO][NF_MAX_HOOKS]; where NPROTO is 32 (maximum protocol families) and NF_MAX_HOOKS is 8 (hook count in 2.6 kernels). For IPv4 (protocol family number 2), the mapping is:

PRE_ROUTING  → nf_hooks[2][0] LOCAL_IN    → nf_hooks[2][1] FORWARD     → nf_hooks[2][2] LOCAL_OUT   → nf_hooks[2][3] POST_ROUTING→ nf_hooks[2][4]

Integration Points in Kernel Source

Several core networking functions invoke NF_HOOK (or its conditional variant) to hand packets to Netfilter:

ip_rcv (net/ipv4/ip_input.c) – entry point for incoming IP packets; uses

NF_HOOK(PF_INET, NF_IP_PRE_ROUTING, skb, dev, NULL, ip_rcv_finish)

.

ip_forward (net/ipv4/ip_forward.c) – processes packets that need to be forwarded; calls

NF_HOOK(PF_INET, NF_IP_FORWARD, skb, skb->dev, rt->u.dst.dev, ip_forward_finish)

.

ip_output (net/ipv4/ip_output.c) – handles locally generated packets; uses

NF_HOOK_COND(PF_INET, NF_IP_POST_ROUTING, skb, NULL, dev, ip_finish_output, !(IPCB(skb)->flags & IPSKB_REROUTED))

.

ip_local_deliver (net/ipv4/ip_input.c) – delivers packets destined for the local host; invokes

NF_HOOK(PF_INET, NF_IP_LOCAL_IN, skb, skb->dev, NULL, ip_local_deliver_finish)

.

ip_push_pending_frames (net/ipv4/ip_output.c) – reassembles fragmented packets for transmission; calls

NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, skb->dst->dev, dst_output)

.

At each hook, Netfilter traverses the corresponding nf_hooks list, executes matching functions, and acts based on the returned verdict.

Summary of the Hook Mechanism

During packet traversal through the kernel’s protocol stack, the five predefined hook points (PRE_ROUTING, LOCAL_IN, FORWARD, LOCAL_OUT, POST_ROUTING) allow registered Netfilter modules to inspect, modify, or drop packets. If no hook is registered, the packet proceeds to the next kernel function via the okfn callback; otherwise, the verdict from the hook determines the subsequent action.

Understanding these hooks, the associated macros, and their placement in the source code is essential for extending or customizing Linux firewall behavior beyond the basic iptables commands.