Why Ping and TCP May Take Different Paths – Understanding ECMP and Routing

Background: Ping vs. TCP

When checking connectivity, we usually run ping. A loss rate of 0% means the destination is reachable, while 100% indicates it is not. The natural question is: if a host can be pinged, will a TCP connection to the same host always succeed? Are the network paths taken by ping (ICMP) and TCP identical?

Short Answer

No, the paths can differ. Even when the underlying network topology does not change, routing decisions may cause ping and TCP packets to travel along different routes.

Why the Paths May Differ

Both ping (ICMP) and TCP packets are built on top of the IP layer, so they share the same physical layer (binary bits on the NIC). However, the routing layer decides where to forward each packet based on the destination IP address and the router’s routing table. If the network environment is unchanged, the routes should be the same, but several factors can cause divergence:

Different routing protocols (static, OSPF, BGP) may generate distinct entries.

When multiple equal‑cost routes exist, routers may employ ECMP (Equal‑Cost Multi‑Path) to balance traffic across them.

Routing Basics

Routers form a graph: each router forwards packets to the next hop according to its routing table . The table is built by protocols such as OSPF, which uses Dijkstra’s algorithm to compute the shortest‑path tree. Each entry contains a destination network, the outgoing interface, and a metric (cost).

When a router receives a packet, it matches the destination IP against the routing table:

Longest prefix match (most specific network).

If prefixes are equal, the entry with the lower administrative distance (e.g., static AD=1 beats OSPF AD=110) wins.

If still tied, the entry with the lower metric (cost) is chosen.

If multiple entries are equal in all respects, the router can enable ECMP, allowing traffic to be split across those paths.

ECMP and Its Effects

ECMP increases aggregate bandwidth by using several parallel links. However, because packets from the same flow may be distributed across different paths, they can arrive out of order. TCP, being a reliable, in‑order protocol, must reorder packets at the receiver. Excessive reordering leads to head‑of‑line blocking, consumes receive‑buffer memory, reduces the effective receive window, and degrades throughput.

To avoid this, routers often hash the five‑tuple (source IP, source port, destination IP, destination port, protocol) to a single hash key, ensuring that all packets of a given flow follow the same path.

Five‑Tuple Differences Explain Path Divergence

Ping uses ICMP, which has no ports, so its five‑tuple differs from TCP’s. Consequently, the hash key for ECMP may map ping and TCP to different paths. Even two separate TCP connections can have different source ports, producing different five‑tuples and potentially different ECMP paths.

Practical Troubleshooting

If you can ping a server but some TCP connections intermittently fail, consider ECMP as a possible cause. You can verify by forcing the source port of a TCP connection using nc: # nc -p 6666 example.com 80 If the connection consistently fails with one source port but succeeds with another, the issue is likely tied to a specific ECMP path.

You can also inspect the receive buffer settings with:

# sysctl net.ipv4.tcp_rmem
net.ipv4.tcp_rmem = 4096(min)    87380(default)  6291456(max)

Summary

Routers generate routing tables via protocols like OSPF (Dijkstra) and select the best path based on prefix length, administrative distance, and metric.

If multiple equal‑cost routes exist, ECMP can split traffic, increasing bandwidth but potentially causing packet reordering.

TCP’s reliability requires in‑order delivery; reordering triggers head‑of‑line blocking and reduces throughput.

ECMP uses a hash of the five‑tuple to keep a flow on a single path, mitigating reordering.

Because ping and TCP have different five‑tuples, they may traverse different routes; even different TCP connections can diverge if source ports differ.

When ping succeeds but TCP intermittently fails, test with fixed source ports (e.g., nc -p) to identify problematic ECMP paths.