Master Linux Routing: From Packets to Containers and Overlay Networks

Why Routing Matters in Container Environments

Containers need to communicate beyond the host, just like pods in Kubernetes, which requires an overlay network built on top of the physical network. Linux acts as a router, forwarding packets between containers and external networks.

When Routing Is Required

Routing is involved both when sending and receiving packets because a host may have multiple network interfaces. The kernel must select the appropriate interface and next‑hop address for each packet.

1.1 Selecting a Route for Outgoing Packets

When a packet reaches the IP layer, the kernel calls ip_queue_xmit, which invokes ip_route_output_ports to look up a route and attach it to the skb. The selected route determines which NIC will transmit the packet.

//file: net/ipv4/ip_output.c
int ip_queue_xmit(struct sk_buff *skb, struct flowi *fl) {
    // routing selection
    // after selection, store route info in skb
    rt = (struct rtable *)__sk_dst_check(sk, 0);
    if (rt == NULL) {
        rt = ip_route_output_ports(...);
        sk_setup_caps(sk, &rt->dst);
    }
    skb_dst_set_noref(skb, &rt->dst);
    ...
    ip_local_out(skb);
}

Linux can have up to 255 routing tables; the default ones are local and main . The ip route list table local command shows entries such as:

# ip route list table local
local 10.143.x.y dev eth0 proto kernel scope host src 10.143.x.y
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1

1.2 Selecting a Route for Incoming Packets

Incoming packets are processed by ip_rcv → ip_rcv_finish. If the packet’s destination matches the local device, it is delivered locally; otherwise, ip_forward forwards it.

//file: net/ipv4/ip_input.c
static int ip_rcv_finish(struct sk_buff *skb) {
    ...
    if (!skb_dst(skb)) {
        int err = ip_route_input_noref(skb, iph->daddr, iph->saddr,
                                      iph->tos, skb->dev);
        ...
    }
    ...
    return dst_input(skb);
}

The helper ip_route_input_noref ultimately calls fib_lookup to find a matching route.

//file: net/ipv4/route.c
int ip_route_input_noref(struct sk_buff *skb, __be32 daddr, __be32 saddr,
                        u8 tos, struct net_device *dev) {
    ...
    res = ip_route_input_slow(skb, daddr, saddr, tos, dev);
    return res;
}

1.3 Where Routing Lives in the Kernel

The routing subsystem is represented by the Forwarding Information Base (FIB). Each network namespace has its own set of fib_table structures, organized in a hash table.

//file: include/net/ip_fib.h
struct fib_table {
    struct hlist_node tb_hlist;
    u32 tb_id;
    int tb_default;
    int tb_num_default;
    unsigned long tb_data[0];
};

By default, only local and main tables exist; enabling CONFIG_IP_MULTIPLE_TABLES allows up to 255 tables.

How Linux Implements Routing

2.1 Routing Tables

Routing tables are stored in struct fib_table and accessed via fib_get_table. The kernel first checks the local table, then the main table; the first match wins.

//file: net/ipv4/route.c
int fib_lookup(struct net *net, const struct flowi4 *flp,
               struct fib_result *res) {
    struct fib_table *table;
    table = fib_get_table(net, RT_TABLE_LOCAL);
    if (!fib_table_lookup(table, flp, res, FIB_LOOKUP_NOREF))
        return 0;
    table = fib_get_table(net, RT_TABLE_MAIN);
    if (!fib_table_lookup(table, flp, res, FIB_LOOKUP_NOREF))
        return 0;
    return -ENETUNREACH;
}

2.2 Routing Lookup Flow

Both sending ( ip_route_output_ports) and receiving ( ip_route_input_slow) eventually call fib_lookup to match the destination against the tables.

Practical Routing Operations

3.1 Enabling Forwarding

By default, Linux does not forward packets that are not destined for the host. Enable forwarding with:

# sysctl -w net.ipv4.ip_forward=1
# sysctl net.ipv4.conf.all.forwarding=1

3.2 Viewing Routing Tables

Check kernel configuration for multiple tables:

# cat /boot/config-$(uname -r) | grep CONFIG_IP_MULTIPLE_TABLES
CONFIG_IP_MULTIPLE_TABLES=y

List the contents of a specific table:

# ip route list table local
local 10.143.x.y dev eth0 proto kernel scope host src 10.143.x.y
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1

The file /etc/iproute2/rt_tables maps numeric IDs to table names.

# cat /etc/iproute2/rt_tables
255 local
254 main
253 default
0 unspec
200 eth0_table

3.3 Modifying Routes

Use route or ip route to add or delete entries. Examples:

Direct host route: # route add -host 192.168.0.100 dev eth0 Host via gateway: # route add -host 192.168.1.100 dev eth0 gw 192.168.0.254 Network route: # route add -net 192.168.1.0/24 dev eth0 Network via gateway: # route add -net 192.168.1.0/24 dev eth0 gw 10.162.132.110 Default route: # route add default gw 192.168.0.1 eth0 For non‑default tables, use ip route with the table option, e.g.:

# ip route add 192.168.5.0/24 via 10.*.*.110 dev eth0 table main

3.4 Testing Routing Rules

Verify a rule with ip route get:

# ip route get 192.168.2.25
192.168.2.25 via 10.*.*.110 dev eth0 src 10.*.*.161 cache

Conclusion

Linux routing is a core component of modern container and cloud‑native networking. Both sending and receiving paths perform route lookups against the local and main tables, selecting the appropriate NIC and next‑hop address. By configuring routing tables and enabling forwarding, containers can seamlessly communicate with external networks.