How to Set Up Dual‑ISP Policy Routing on Linux and FreeBSD with Nginx

Why Policy Routing Is Needed

In many Internet companies, servers hosted in IDC environments use two ISP lines (e.g., China Telecom and China Unicom). Without policy routing, traffic from each ISP may exit through the wrong interface, causing slow responses or unreachable services. Proper policy routing ensures that packets enter and leave through the same ISP interface, achieving source‑in‑source‑out behavior.

Linux (CentOS 6.5) Configuration

Assume three IP addresses: Telecom 58.215.55.100/24 (eth1), Unicom 117.121.132.100/24 (eth2), and an internal 192.168.1.100/24 (eth0). The following steps configure persistent policy routing using the standard route‑ethX and rule‑ethX mechanisms.

echo "default via 58.215.55.1 src 58.215.55.100 table telcom" > /etc/sysconfig/network-scripts/route-eth1

echo "from 58.215.55.100 table telcom" > /etc/sysconfig/network-scripts/rule-eth1

echo "default via 117.121.132.1 src 117.121.132.100 table unicom" > /etc/sysconfig/network-scripts/route-eth2

echo "from 117.121.132.100 table unicom" > /etc/sysconfig/network-scripts/rule-eth2

Add the custom routing tables:

echo "250 telcom" >> /etc/iproute2/rt_tables

echo "251 unicom" >> /etc/iproute2/rt_tables

Restart the network service to apply the rules: /etc/init.d/network restart With the policy routes active, Nginx can listen on all three IPs using a simple listen 80; directive; the routing tables will direct traffic correctly.

FreeBSD 10.3 Configuration

FreeBSD does not support multiple routing tables out of the box, so the kernel must be rebuilt with the ROUTETABLES option.

Download and extract the source tree:

fetch http://mirrors.sohu.com/FreeBSD/amd64/10.3-RELEASE/src.txz

tar zxf src.txz -C /

Enable multiple routing tables:

cd /usr/src/sys/amd64/conf

cp GENERIC /root/ROUTES

ln -s /root/ROUTES

echo "options ROUTETABLES=16" >> ROUTES

Rebuild and install the new kernel:

cd /usr/src

make NO_MODULES=1 kernel KERNCONF=ROUTES KODIR=/boot/routes

mv /boot/kernel/kernel /boot/kernel/kernel.bak

cp /boot/routes/kernel /boot/kernel/

Enable the PF firewall (or ipfw) in /etc/rc.conf:

echo 'pf_enable="YES"' >> /etc/rc.conf

echo 'pf_rules="/etc/pf.conf"' >> /etc/rc.conf

Configure the two external IPs in /etc/rc.conf (the internal IP is already set):

echo 'ifconfig_vtnet1="inet 58.215.55.100/24"' >> /etc/rc.conf

echo 'ifconfig_vtnet2="inet 117.121.132.100/24"' >> /etc/rc.conf

Create two additional routing tables (default table 0 already points to the internal gateway):

echo "/usr/sbin/setfib 1 /sbin/route add default 58.215.55.1" >> /etc/rc.local

echo "/usr/sbin/setfib 2 /sbin/route add default 117.121.132.1" >> /etc/rc.local

Add PF rules to bind outgoing traffic to the correct interface:

echo "pass out quick route-to (vtnet1 58.215.55.1) from 58.215.55.100 to any" >> /etc/pf.conf

echo "pass out quick route-to (vtnet2 117.121.132.1) from 117.121.132.100 to any" >> /etc/pf.conf

Reboot the system to load the new kernel and apply the PF rules.

After the FreeBSD setup, Nginx must use the setfib parameter in its listen directives to select the appropriate routing table:

listen 58.215.55.100:80 setfib=1;

listen 117.121.132.100:80 setfib=2;

listen 192.168.1.100:80 setfib=0;

Also change the event method from epoll to kqueue for FreeBSD compatibility.

Additional Notes

When using HAProxy on FreeBSD, the same multi‑line principle applies: create separate configuration files (e.g., haproxy.cfg.telcom, haproxy.cfg.unicom, haproxy.cfg.inside) with distinct bind addresses, and adjust the startup script to launch the appropriate instance based on the routing table.