Master Linux Network Routing & Forwarding: From Theory to Real-World Practice

Linux Network Routing and Forwarding: A Complete Guide

In the cloud‑native era, mastering Linux routing is essential for operations engineers; it underpins high‑availability clusters, load balancing, and multi‑layer network architectures.

1. Why routing matters

Imagine a recent infrastructure upgrade that turned a single network into multiple subnets and VLANs, only to discover a critical service unreachable due to a routing error—a common scenario in production.

Quickly locate network faults : 80% of network issues are routing‑related

Optimize network performance : proper routing can improve efficiency by over 30%

Build complex network architectures : enable cross‑subnet communication, VPN tunnels, load balancing, and more

Boost career competitiveness : a key skill distinguishing junior from senior ops engineers

2. Core principles of Linux routing

2.1 The nature of the routing table

The kernel maintains a routing table that maps destination IPs to next hops. When a packet is sent, the kernel performs the following steps:

Destination address match : check the target IP address

Route lookup : find the most specific entry in the routing table

Forwarding decision : decide which interface to use

Next‑hop handling : determine the next‑hop address or deliver locally

Linux defines multiple routing tables in /etc/iproute2/rt_tables:

# cat /etc/iproute2/rt_tables
255   local
254   main
253   default
0     unspec

2.2 Longest Prefix Match algorithm

Linux selects the route with the longest matching prefix. Example entries:

10.0.0.0/8 via 192.168.1.1

10.1.0.0/16 via 192.168.1.2

10.1.1.0/24 via 192.168.1.3

When sending to 10.1.1.100, the /24 entry is chosen because it has the longest prefix.

2.3 Kernel route cache

To improve performance, the kernel caches route lookups. Commands to view and flush the cache:

# ip -s route show cache
# ip route flush cache
# watch -n 1 'cat /proc/net/stat/rt_cache'

3. Hands‑on: Basic routing configuration

3.1 Static routes

Scenario: Server A needs to reach Server B in a different subnet.

#!/bin/bash
# View current routes
echo "=== Current routing ==="
ip route show

# Add static route
sudo ip route add 10.20.0.0/24 via 192.168.1.254 dev eth0

# Verify
ip route get 10.20.0.100
ping -c 3 10.20.0.100

# Add default route
sudo ip route add default via 192.168.1.1

# Persist on CentOS/RHEL
cat >> /etc/sysconfig/network-scripts/route-eth0 <<EOF
10.20.0.0/24 via 192.168.1.254 dev eth0
EOF

# Persist on Ubuntu/Debian
cat >> /etc/netplan/01-network.yaml <<EOF
network:
  version: 2
  ethernets:
    eth0:
      routes:
        - to: 10.20.0.0/24
          via: 192.168.1.254
EOF
netplan apply

3.2 Dynamic routing (OSPF example)

#!/bin/bash
# Install FRR (OSPF implementation)
sudo apt-get update
sudo apt-get install -y frr frr-pythontools

# Enable OSPF daemon
sudo sed -i 's/ospfd=no/ospfd=yes/' /etc/frr/daemons
sudo systemctl restart frr

# Configure OSPF
sudo vtysh -c "
configure terminal
router ospf
  ospf router-id 1.1.1.1
  network 192.168.1.0/24 area 0
  network 10.0.0.0/24 area 0
  passive-interface eth0
exit
write
"

4. Advanced: Policy routing

4.1 Multiple routing tables

#!/bin/bash
# Create custom tables
echo "100 isp1" >> /etc/iproute2/rt_tables
echo "200 isp2" >> /etc/iproute2/rt_tables

# Add routes for each ISP
ip route add default via 10.1.1.1 dev eth1 table isp1
ip route add 192.168.0.0/24 dev eth0 table isp1
ip route add default via 10.2.2.1 dev eth2 table isp2
ip route add 192.168.0.0/24 dev eth0 table isp2

# Policy rules based on source subnet
ip rule add from 192.168.1.0/24 table isp1 priority 100
ip rule add from 192.168.2.0/24 table isp2 priority 200

# Policy based on port (HTTP)
ip rule add fwmark 0x1 table isp1
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 0x1

4.2 Load‑balancing routes

# Add ECMP default route
ip route add default \
  nexthop via 10.1.1.1 dev eth1 weight 1 \
  nexthop via 10.2.2.1 dev eth2 weight 2

# ECMP for a specific subnet
ip route add 172.16.0.0/16 \
  nexthop via 192.168.1.1 \
  nexthop via 192.168.2.1

5. IP forwarding and NAT

5.1 Enable IP forwarding

# Temporary enable
echo 1 > /proc/sys/net/ipv4/ip_forward

# Permanent enable
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
sysctl -p

5.2 NAT configuration

# Define interfaces
WAN_IFACE="eth0"
LAN_IFACE="eth1"
LAN_NETWORK="192.168.100.0/24"

# SNAT (source NAT)
iptables -t nat -A POSTROUTING -s $LAN_NETWORK -o $WAN_IFACE -j MASQUERADE

# DNAT (port forwarding) – expose internal web server
iptables -t nat -A PREROUTING -i $WAN_IFACE -p tcp --dport 80 -j DNAT --to-destination 192.168.100.10:8080

6. Troubleshooting and performance tuning

6.1 Common routing diagnostics

#!/bin/bash
echo "=== Routing diagnosis start ==="
ip link show
ip addr show
ip route show
ip route | grep default
gw=$(ip route | grep default | awk '{print $3}')
ping -c 2 -W 2 $gw
nslookup google.com
traceroute -n 8.8.8.8 -m 10
ip neigh show
iptables -L -n | head -20

6.2 Performance monitoring

#!/bin/bash
echo "=== Interface traffic ==="
sar -n DEV 1 5

echo "=== Route cache stats ==="
cat /proc/net/stat/rt_cache

echo "=== Conntrack stats ==="
conntrack -S

echo "=== Socket buffer usage ==="
ss -m

echo "=== TCP connection stats ==="
ss -s

7. Real‑world case: Enterprise‑grade routing

7.1 Multi‑data‑center interconnect

#!/bin/bash
setup_dc_routing() {
  local DC_NAME=$1 LOCAL_NET=$2 REMOTE_NET=$3 TUNNEL_IP=$4 REMOTE_IP=$5
  echo "Configuring $DC_NAME data‑center routing..."
  ip tunnel add gre1 mode gre remote $REMOTE_IP local $TUNNEL_IP
  ip addr add 172.16.0.1/30 dev gre1
  ip link set gre1 up
  ip route add $REMOTE_NET dev gre1
  cat > /etc/frr/ospfd.conf <<EOF
router ospf
  ospf router-id $TUNNEL_IP
  network $LOCAL_NET area 0
  network 172.16.0.0/30 area 0
EOF
  ip link set gre1 mtu 1400
  ethtool -K gre1 gso off
  systemctl restart frr
}
setup_dc_routing "Beijing" "10.1.0.0/16" "10.2.0.0/16" "1.1.1.1" "2.2.2.2"

7.2 High‑availability routing (VRRP)

#!/bin/bash
apt-get install -y keepalived
cat > /etc/keepalived/keepalived.conf <<'EOF'
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass secret123
    }
    virtual_ipaddress {
        192.168.1.254/24
    }
    track_script {
        check_gateway
    }
}

vrrp_script check_gateway {
    script "/usr/local/bin/check_gateway.sh"
    interval 2
    weight -20
}
EOF

cat > /usr/local/bin/check_gateway.sh <<'EOF'
#!/bin/bash
ping -c 1 -W 1 8.8.8.8 > /dev/null 2>&1
exit $?
EOF
chmod +x /usr/local/bin/check_gateway.sh
systemctl restart keepalived

8. Security hardening for routing

8.1 Prevent route spoofing

# Enable reverse‑path filtering on all interfaces
for i in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 1 > $i; done

# Disable source routing
echo 0 > /proc/sys/net/ipv4/conf/all/accept_source_route

# Disable ICMP redirects
echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects

# Drop private‑source packets arriving on external interface
iptables -A INPUT -s 10.0.0.0/8 -i eth0 -j DROP
iptables -A INPUT -s 172.16.0.0/12 -i eth0 -j DROP
iptables -A INPUT -s 192.168.0.0/16 -i eth0 -j DROP

# Rate‑limit ICMP echo requests
iptables -A INPUT -p icmp --icmp-type echo-request -m limit --limit 1/s --limit-burst 3 -j ACCEPT
iptables -A INPUT -p icmp --icmp-type echo-request -j DROP

8.2 Route monitoring and alerts

#!/bin/bash
ALERT_EMAIL="[email protected]"
ROUTE_BASELINE="/var/lib/route_baseline"

ip route show > ${ROUTE_BASELINE}.new
if [ -f ${ROUTE_BASELINE} ]; then
  diff ${ROUTE_BASELINE} ${ROUTE_BASELINE}.new > /tmp/route_diff
  if [ -s /tmp/route_diff ]; then
    echo "Route table changed:" | mail -s "Route Alert" $ALERT_EMAIL < /tmp/route_diff
    logger -t "route-monitor" "Route table changed"
    cat /tmp/route_diff | logger -t "route-monitor"
  fi
fi
mv ${ROUTE_BASELINE}.new ${ROUTE_BASELINE}

9. Container networking

#!/bin/bash
setup_container_routing() {
  # Create custom bridge network
  docker network create --driver bridge \
    --subnet=172.20.0.0/16 \
    --gateway=172.20.0.1 \
    --opt "com.docker.network.bridge.name"="docker-custom" \
    custom-net

  # Example inter‑bridge route
  ip route add 172.21.0.0/16 via 172.20.0.2

  # Enable NAT for containers
  iptables -t nat -A POSTROUTING -s 172.20.0.0/16 ! -o docker-custom -j MASQUERADE

  # Port mapping example
  iptables -t nat -A DOCKER -p tcp --dport 8080 -j DNAT --to-destination 172.20.0.10:80
}

10. Summary and next steps

Key takeaways

Routing fundamentals : understand Linux routing tables and longest‑prefix‑match algorithm

Configuration skills : static, dynamic, and policy routing

NAT and forwarding : set up IP forwarding and network address translation

Troubleshooting : quickly locate and resolve routing problems

Performance optimization : monitor and tune routing performance

Security hardening : protect routing infrastructure from attacks

Advanced learning path

Deep dive into BGP, the backbone protocol of the Internet

Study Software‑Defined Networking (SDN) technologies

Master eBPF for high‑performance network programming

Learn Kubernetes networking, CNI plugins, and service mesh concepts

Automate network configuration with Ansible, Terraform, or other IaC tools