Using mtr to Pinpoint Network Packet‑Loss Nodes: A Step‑by‑Step Guide

When a user experiences slow network, timeouts, or request failures, the first mistake is to blame a hop solely because mtr shows a non‑zero Loss% column. mtr combines traceroute path discovery with repeated ping statistics and is useful for narrowing down suspicious links, but it does not directly measure TCP‑level loss.

1. Define investigation scope

Record the business protocol, target domain/IP, port, fault window, source network, any proxy/VPN, and whether other services on the same host are abnormal. Use the same source, destination, protocol, and port for both fault‑time and normal‑time runs.

#!/usr/bin/env bash
set -euo pipefail
command -v mtr
mtr --version
mtr --help | sed -n '1,120p'
ip -brief address

If mtr is missing, install the distro‑provided package (mtr or mtr‑tiny) via the controlled repository; never download binaries from unknown sources.

2. Resolve target address and routing

Check whether the domain resolves to multiple A/AAAA records or load‑balanced addresses, and note which address mtr actually probes.

TARGET="<target domain>"
getent ahosts "$TARGET"
getent ahostsv6 "$TARGET" || true
ip route get <target IP>
ip rule show

Use ip route get to see the real source address, next hop, and output interface; on hosts with multiple NICs, policy routing, containers, or VPNs the default route may not represent the business path.

3. Generate a baseline report

Interactive mode is good for on‑site observation, but report mode ( -r) is needed for saving and later comparison. Increase the probe count ( -c) for statistical stability, remembering that more probes increase load and duration.

mtr -r -w -z -c 100 <target IP>

Interpret the Loss%, Avg, Best, Wrst, and StDev columns together. A single hop with high loss followed by clean downstream hops usually indicates rate‑limiting of ICMP replies, not actual forwarding loss.

4. Align probing protocol with the business flow

When ICMP is throttled, use TCP or UDP probes that match the service (e.g., TCP 443 for HTTPS). mtr does not complete the application handshake; it only sends probe packets.

mtr -r -w -z -c 100 --tcp --port 443 <target IP>
mtr -r -w -z -c 100 --udp --port <port> <target IP>

Confirm that the installed mtr version supports the long options; firewall rejections of TCP probes still yield path information and should not be equated with service unavailability.

5. Bind to a specific interface or source address

On multi‑exit hosts, use -I to select the interface and -a to bind the source address. If the command fails, inspect policy routing and security rules instead of forcing the bind.

mtr -r -w -c 100 -I <iface> -a <source IP> --tcp --port <port> <target IP>

Record NIC RX/TX error, dropped, and carrier counters before and after the fault; use the delta, not the absolute value.

6. Distinguish real loss from apparent loss

If a hop shows high loss but later hops are near zero, the loss is likely due to the device deprioritising ICMP replies, not forwarding loss. Consistent loss from a hop to the final destination, corroborated by repeated tests and business‑layer errors, warrants deeper investigation.

7. Export reports for automation

mtr -r -c 100 --json <target IP> > <outdir>/mtr.json
mtr -r -c 100 --csv  <target IP> > <outdir>/mtr.csv
mtr -r -c 100 --xml  <target IP> > <outdir>/mtr.xml

Save the command line, timestamp, source address, and target IP alongside the report; screenshots alone lose essential fields.

8. Multi‑target and time‑window comparison

Run the same probe against several targets (same rack, same region, business dependency, final destination) to see whether the issue is local to the source or shared across paths.

#!/usr/bin/env bash
set -euo pipefail
OUT_DIR="<report dir>"
SOURCE="<source IP>"
PORT="<port>"
TARGETS=("<rack IP>" "<region IP>" "<business IP>")
install -d -m 0700 "$OUT_DIR"
for target in "${TARGETS[@]}"; do
  stamp=$(date +%Y%m%d-%H%M%S)
  mtr -r -w -z -c 100 -a "$SOURCE" --tcp --port "$PORT" "$target" > "$OUT_DIR/mtr-$target-$stamp.txt"
done

Compare the files with diff -u; the diff shows textual differences, not statistical significance, but helps locate changed hops or loss trends.

9. Reverse‑direction verification

Network paths are often asymmetric. If you have access to the remote side, run the same mtr command from there with identical protocol, port, and count to confirm the reverse path.

mtr -r -w -z -c 100 --tcp --port <port> <source host IP>

10. Check MTU and fragmentation

When large packets fail but small ones succeed, investigate path MTU using tracepath and sized pings.

tracepath -n <target IP>
ping -n -M do -s 1472 -c 3 <target IP>

11. Gather host‑side counters

Collect NIC statistics, softnet counters, TCP retransmissions, and listen queue overflows before and after the incident.

ip -s link show dev <iface>
ethtool -S <iface> | grep -Ei 'drop|err|miss|timeout|reset' || true
cat /proc/net/softnet_stat
mpstat -P ALL 1 5
nstat -az | grep -E 'TcpRetransSegs|TcpExtListenOverflows|TcpExtListenDrops|TcpTimeouts|IpInDiscards'
ss -s

12. Review firewall and security policies

List nftables, iptables, and ip6tables rules without modifying them.

sudo nft list ruleset
sudo iptables -S
sudo ip6tables -S

13. Capture matching traffic

Use tcpdump with the same protocol and ports to verify that probes actually leave the host and responses are received.

sudo tcpdump -i <iface> -nn -vv '(icmp or tcp port <port> or udp port <port>) and host <target IP>'

14. Assemble an upgradeable evidence package

When the investigation points to the network team or cloud provider, the ticket should include source address, region, target IP, start/end timestamps, mtr version, full command line, protocol/port, loss trend, reverse‑probe results, business error rate, interface counters, routing info, and relevant PCAP files.

#!/usr/bin/env bash
set -euo pipefail
TARGET="<target IP>"
SOURCE="<source IP>"
INTERFACE="<iface>"
PORT="<port>"
OUT_DIR="<report dir>/incident-$(date +%Y%m%d-%H%M%S)"
install -d -m 0700 "$OUT_DIR"
date -Is > "$OUT_DIR/time.txt"
ip route get "$TARGET" from "$SOURCE" > "$OUT_DIR/route.txt"
ip -s link show dev "$INTERFACE" > "$OUT_DIR/link.txt"
nstat -az > "$OUT_DIR/nstat.txt"
mtr -r -w -z -c 100 -a "$SOURCE" --tcp --port "$PORT" "$TARGET" > "$OUT_DIR/mtr.txt"
curl -fsS --connect-timeout 3 --max-time 15 "https://<target domain>/<health path>" > "$OUT_DIR/health.txt"

The script writes all evidence to a timestamped directory without altering network configuration or automatically uploading data.