Network Unreachable? Layered Troubleshooting with ping, telnet, curl & traceroute

Define the Failure: Source, Destination, Protocol

Before troubleshooting, fix five elements: source host or Pod, target domain/IP, target port, transport protocol, and failure time. Confirm whether all sources fail or only one, all ports or a specific service, continuous or intermittent. Record timestamp, host identity, kernel version, and network namespace; container localhost, routing, and firewall may differ from the host.

date --iso-8601=seconds
hostnamectl
uname -r
systemd-detect-virt --container || true
readlink /proc/self/ns/net

If the issue originates in a container, run tests inside the same network namespace. A successful curl on the host only proves the host path works, not the Pod's DNS, NetworkPolicy, service mesh, or source NAT.

What Each Tool Actually Verifies

ping – ICMP Echo, round-trip latency, basic reachability. Success means the target or an intermediate device replied ICMP and the bidirectional path was usable at test time. Failure does not prove TCP port closed, host down, or DNS broken.

telnet (or nc) – TCP connection establishment and optional plaintext interaction. Success proves the three-way handshake to the given IP:port succeeded. Failure does not prove TLS/HTTP or UDP health.

curl – DNS, TCP, TLS, HTTP/application response. Success proves the full URL-to-application chain completed under the specified conditions. Failure does not imply ping must succeed or that all clients are affected.

traceroute – per-hop TTL/Hop Limit feedback. Success shows some routers returned ICMP Time Exceeded. Asterisks (*) do not mean that router drops business traffic, nor that the displayed path is symmetric.

Recommended order is not mechanical: if the symptom is “HTTPS returns 502”, start with curl -v, service logs, and reverse proxy; if the target IP is unknown, check DNS first; if no default route exists, skip public traceroute.

Step Zero: Eliminate Local Basics

Interface, Address, and Link Status

ip -br link

and ip -br address give a quick overview; ip -s link show dev <interface> shows error and drop counters. ethtool <interface> reports speed, duplex, and link detection. A physical NIC showing Link detected: no requires checking cables, switch ports, bond members, and drivers. Virtual interfaces may not support all ethtool fields. RX dropped increments can stem from driver, queue, or stack – not necessarily external link loss.

Let the Kernel Show the Actual Route

Don't rely only on the default route. Policy routing, multiple tables, source address selection, and VRF can change the real path. ip route get <targetIP> simulates a lookup under current rules; adding from <sourceIP> uses the actual source address the application binds or the pre-SNAT address. If the output interface, next hop, or src differs from expectations, fix routing/policy first – don't guess from ping. Route changes are high-risk: keep existing config, confirm out-of-band access, and validate with temporary routes before persisting.

ip rule show
ip route show table main
ip route get <targetIP>
ip route get <targetIP> from <sourceIP>

Check Neighbor Resolution on the Same Segment

IPv4 ARP and IPv6 NDP appear in the neighbor table. FAILED or prolonged INCOMPLETE for the gateway or same-segment target indicates layer-2 resolution failure.

ip neigh show dev <interface>
ping -c 3 -W 1 <gatewayIP>
ip neigh show to <gatewayIP> dev <interface>

If the gateway is reachable but the remote host isn't, layer-2 and the path to the gateway are healthy. If the gateway also doesn't reply, remember it may block ICMP; combine neighbor state and packet captures. Avoid ip neigh flush all – it disrupts existing connections. If a single stale entry must be cleared, verify impact first.

Separate DNS Resolution Success from Correctness

ping <domain>

failure may occur before DNS. Inspect system resolver config, then compare the system resolver (via NSS) with a direct query to a specific DNS server.

cat /etc/resolv.conf
getent ahosts <targetDomain>
resolvectl query <targetDomain> 2>/dev/null || true
dig +time=2 +tries=1 <targetDomain> A
dig +time=2 +tries=1 <targetDomain> AAAA

getent

follows Name Service Switch, closer to application behavior; dig queries the server in /etc/resolv.conf directly, bypassing mDNS, LDAP, or local caches. With systemd-resolved, /etc/resolv.conf may point to a local stub; resolvectl shows per-interface DNS and search domains. To test a specific DNS server:

dig @<dnsServerIP> <targetDomain> A +noall +answer +comments
dig @<dnsServerIP> <targetDomain> AAAA +noall +answer +comments
dig +trace <targetDomain>

dig +trace

iterates from root; it may be blocked by egress policy and does not reflect internal split-horizon views. DNS success requires correct RCODE, answer section, TTL, and addresses belonging to the current environment – not merely “an IP returned”.

ping: Verify ICMP Reachability and Quality

Test Domain and IP Separately

First ping the IP to exclude DNS; then ping the domain to observe the actual address family and address chosen.

ping -c 4 -W 2 <targetIP>
ping -4 -c 4 -W 2 <targetDomain>
ping -6 -c 4 -W 2 <targetDomain>

In iputils, -c is count, -W is per-reply timeout (seconds). BusyBox options may differ; run ping --help. Forcing IPv4/IPv6 reveals dual-stack issues like “AAAA resolves but IPv6 egress missing”.

Example output (illustrative):

64 bytes from 192.0.2.20: icmp_seq=1 ttl=56 time=12.4 ms
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 11.9/12.3/12.8/0.3 ms

Success means the target replied to ICMP Echo and the return path works. TTL can hint at path changes but initial TTL varies by OS, so exact hop count or OS inference is unreliable. A short 0% loss sample doesn't prove absence of jitter under load.

Fix Source Interface or Address

Multi-homed hosts may send from the wrong egress. Use -I to specify interface or source address, matching the application's binding.

ping -I <interface> -c 4 -W 2 <targetIP>
ping -I <sourceIP> -c 4 -W 2 <targetIP>
ip route get <targetIP> from <sourceIP>

If interface-specific test fails while default succeeds, common causes: missing route on that interface, absent policy routing, incorrect return route, or upstream source-address filtering. Conclusions need routing lookup and packet capture support.

Detect MTU/PMTU Black Holes

Small packets pass, large packets time out – typical during TLS handshake or large response transfers. Use ping -M do -s <payload> to set IPv4 DF bit and forbid local fragmentation. For Ethernet MTU 1500, IPv4 ICMP payload of 1472 bytes (20B IPv4 + 8B ICMP headers) is a starting point; IPv6 header length differs.

ping -4 -M do -s 1472 -c 3 -W 2 <targetIP>
ping -4 -M do -s 1400 -c 3 -W 2 <targetIP>
tracepath <targetIP>
ip link show dev <interface>

If 1400 works but 1472 fails, bisect payload size and investigate tunnels, VPNs, cloud encapsulation, or filtered ICMP Fragmentation Needed. Don't globally lower MTU to 1400 – it affects all connections on the interface. Confirm scope, redundancy, rollback value; test on a canary interface first, then re-test large packets and business TLS.

Correct Interpretation of ping Failures

Name or service not known

: DNS/NSS issue, not ICMP. Network is unreachable: No matching route on source; check addresses, routes, policies. Destination Host Unreachable: Note whether the message originates from source, gateway, or intermediate router; source-originated often accompanies neighbor failure.

All timeouts: Could be silent drop, target blocking ICMP, missing return path, or target down – cannot assign blame solely from timeout.

High loss/latency: Use longer controlled samples, correlate with mtr, interface counters, and business hours; ICMP may be rate-limited, so loss may not reflect TCP loss.

telnet: Only Verifies TCP Handshake, Not Service Health

Though nc is better for automation, telnet <host> <port> remains common for quick distinction among timeout, refused, and connected. Don't use it for UDP; never send passwords, tokens, or production secrets in cleartext sessions. telnet <targetIP> <targetPort> Common outcomes: Connected to ...: TCP three-way handshake complete. Press Ctrl+] then quit to exit.

Immediate Connection refused: Usually TCP RST received – path largely reachable, but port not listening or firewall actively rejects.

Long timeout: SYN or SYN-ACK dropped, return path broken, or target unresponsive – need packet capture. No route to host: Could be local routing error or firewall returning ICMP prohibited; combine ip route get with capture.

Automation prefers netcat. Implementations differ (OpenBSD nc, Nmap ncat, BusyBox nc); run nc -h first.

nc -vz -w 3 <targetIP> <targetPort>
timeout 5 bash -c '</dev/tcp/<targetIP>/<targetPort>'

The second uses Bash's /dev/tcp feature, not POSIX sh, and gives limited diagnostics. Both only verify TCP, not TLS certificates, HTTP status, or response content.

Confirm Listening on the Server Side

If you have target server access, check port, bind address, and process. A service bound only to 127.0.0.1 succeeds locally but fails remotely; IPv6-only bind behavior depends on net.ipv6.bindv6only and socket options.

ss -lntp 'sport = :<targetPort>'
systemctl status <serviceName> --no-pager
systemctl show <serviceName> -p MainPID -p ActiveState -p SubState
journalctl -u <serviceName> --since '-30 minutes' -p warning --no-pager

ss -p

for other users' processes usually needs root. If nothing listens, check service startup failure, config port, and process logs – don't open firewall to fix “process not listening”.

Test loopback and host address separately to isolate application binding from external path:

curl --noproxy '*' --connect-timeout 3 http://127.0.0.1:<targetPort>/
curl --noproxy '*' --connect-timeout 3 http://<localBusinessIP>:<targetPort>/
nc -vz -w 3 <localBusinessIP> <targetPort>

If the service isn't HTTP, don't force curl; use protocol-appropriate health checks. --noproxy '*' prevents environment proxies from redirecting “local test” to a proxy server.

Use Packet Capture to Locate Timeout

Capture involves business metadata (addresses, domains, unencrypted content); obey permission/retention policies, limit interface, host, port, packet count, and duration. No long-term unfiltered production captures.

sudo timeout 20 tcpdump -ni <interface> \
  -c 100 'host <targetIP> and tcp port <targetPort>' \
  -w /var/tmp/connectivity-check.pcap

sudo tcpdump -nn -tttt -r /var/tmp/connectivity-check.pcap

Interpretation: Repeated client SYN with no SYN-ACK means request left capture point but response didn't return; immediate RST means port refused; handshake completes but client sends request and server doesn't reply application data – focus on server process, TLS, or upstream. Single-end capture can't pinpoint which hop drops; if needed, capture simultaneously on server and align timestamps.

curl: Segment Timing for DNS, TCP, TLS, HTTP

curl -v

shows resolution, connection, TLS, and request/response but may log sensitive headers (Authorization, Cookie). Prefer non-sensitive health endpoints; sanitize before saving.

curl --noproxy '*' -v \
  --connect-timeout 3 \
  --max-time 10 \
  https://<targetDomain>:<targetPort>/health

--connect-timeout

limits connection phase (includes DNS, TCP, TLS for HTTPS); --max-time limits entire transfer. --noproxy '*' forces direct connection, bypassing HTTP_PROXY, HTTPS_PROXY, NO_PROXY. If traffic must go through a proxy, test both direct and proxy paths separately – don't treat direct result as final path.

Output Comparable Phase Timings

curl -w

turns “feels slow” into per-phase durations. remote_ip confirms which resolved address was actually used.

curl --noproxy '*' -sS -o /dev/null \
  --connect-timeout 3 --max-time 15 \
  -w 'remote_ip=%{remote_ip}
http_code=%{http_code}
namelookup=%{time_namelookup}
connect=%{time_connect}
appconnect=%{time_appconnect}
starttransfer=%{time_starttransfer}
total=%{time_total}
' \
  https://<targetDomain>:<targetPort>/health

Judge by differences: time_connect - time_namelookup ≈ TCP connection phase. time_appconnect - time_connect ≈ TLS phase. time_starttransfer - time_appconnect mainly request send, server processing, and first-byte wait. For plain HTTP, time_appconnect is usually 0.

A single sample isn't a trend; repeat at controlled frequency and keep timestamps.

Continuous sampling script (CSV output, retains failures to avoid survivor bias):

#!/usr/bin/env bash
set -uo pipefail

URL="https://<targetDomain>:<targetPort>/health"
COUNT=10
INTERVAL=2

printf 'timestamp,exit_code,remote_ip,http_code,dns,tcp,tls,ttfb,total
'
for ((i=1; i<=COUNT; i++)); do
  stamp="$(date --iso-8601=seconds)"
  metrics="$(curl --noproxy '*' -sS -o /dev/null \
    --connect-timeout 3 --max-time 10 \
    -w '%{remote_ip},%{http_code},%{time_namelookup},%{time_connect},%{time_appconnect},%{time_starttransfer},%{time_total}' \
    "${URL}")"
  rc=$?
  printf '%s,%d,%s
' "${stamp}" "${rc}" "${metrics}"
  sleep "${INTERVAL}"
done

No set -e so sampling continues after a failure; set -u and pipefail retained. Curl may output partial metrics on failure, so always read exit_code.

Bypass DNS While Keeping Host and SNI

Direct https://<targetIP>/ changes TLS SNI and HTTP Host, possibly causing wrong certificate or default vhost. --resolve fixes the connection IP while preserving domain semantics.

curl --noproxy '*' -v \
  --resolve '<targetDomain>:<targetPort>:<targetIP>' \
  --connect-timeout 3 --max-time 10 \
  https://<targetDomain>:<targetPort>/health

If normal request fails but --resolve succeeds, evidence points to DNS result, cache, or address-family selection. If both fail identically after TLS, investigate server side. --resolve affects only this curl invocation, not system hosts – safe for verification.

Separate IPv4 and IPv6

Dual-stack domains may break only one family. Force each and record actual remote IP:

curl -4 --noproxy '*' -sS -o /dev/null \
  -w 'ip=%{remote_ip} code=%{http_code} total=%{time_total}
' \
  --connect-timeout 3 --max-time 10 https://<targetDomain>/

curl -6 --noproxy '*' -sS -o /dev/null \
  -w 'ip=%{remote_ip} code=%{http_code} total=%{time_total}
' \
  --connect-timeout 3 --max-time 10 https://<targetDomain>/

If only IPv6 fails, check IPv6 address, default route, neighbor discovery, firewall, and upstream return path. Don't hastily delete AAAA records or disable IPv6 globally – that impacts other services. Temporary -4 validation stops at client layer, not infrastructure fix.

TLS Failures: Check Certificate Chain, Name, and Time

curl

reporting certificate verify failed ≠ TCP unreachable. Causes: expired cert, wrong system time, missing intermediate, SNI mismatch, or broken local CA bundle. Never use curl -k as a fix; it skips verification, only for brief diagnosed comparison without sensitive credentials.

date -u
timedatectl status
openssl s_client \
  -connect <targetDomain>:<targetPort> \
  -servername <targetDomain> \
  -verify_return_error </dev/null

openssl s_client

output can be long. Check subjectAltName, validity, issuer, chain verification return code, and whether server sends full intermediate chain. Root cause must be supported by cert content and verification error – don't lump all TLS errors as “certificate expired”.

Extract key cert fields:

openssl s_client -connect <targetDomain>:<targetPort> \
  -servername <targetDomain> -showcerts </dev/null 2>/dev/null | \
openssl x509 -noout -subject -issuer -dates -ext subjectAltName

Pipeline parses only the first server cert; full chain still needs s_client verification output. If -ext unsupported, check openssl x509 -help and use -text separately.

HTTP Status Is Not a Network-Layer Failure

200

usually means end-to-end application success. 301/302 may redirect to wrong domain. 401/403 indicates arrival at application or gateway auth layer. 404 may be Host/path mismatch. 502/503/504 often relate to reverse proxy upstream or capacity.

Use response headers and limited body to locate; avoid downloading large files.

curl --noproxy '*' -sS -D /var/tmp/response.headers \
  -o /var/tmp/response.body \
  --range 0-4095 \
  --connect-timeout 3 --max-time 10 \
  https://<targetDomain>:<targetPort>/health

sed -n '1,40p' /var/tmp/response.headers
wc -c /var/tmp/response.body

--range

only limits if server supports Range; diagnostic endpoints should return small responses. Response files may contain sensitive data – check permissions and clean per policy. Don't misreport 503 as “port unreachable”; TCP/TLS likely succeeded.

Proxy Variables Often Cause “Only This User Fails”

systemd services, interactive shells, containers may have different proxy settings. Read-only check of variables and their sources:

env | grep -iE '^(http|https|all|no)_proxy='
systemctl show <serviceName> -p Environment
curl -v --connect-timeout 3 --max-time 10 https://<targetDomain>/ 2>&1 | \
  grep -E 'Uses proxy env variable|Connected to|Trying '

Proxy URLs may contain credentials – sanitize before display or ticket submission. Compare curl --noproxy '*' with normal curl: only proxy path fails → check proxy DNS, ACL, auth, and NO_PROXY matching; don't globally delete proxy variables and break package managers or other services.

traceroute: Locate Path Boundaries, Don't Treat Asterisks as Breakpoints

Classic traceroute sends probes with increasing TTL; routers returning ICMP Time Exceeded reveal the hop. Default probe type varies (Linux often UDP); networks may allow only ICMP or TCP, so match business protocol.

traceroute -n -w 2 -q 1 <targetIP>
traceroute -I -n -w 2 -q 1 <targetIP>
traceroute -T -p <targetPort> -n -w 2 -q 1 <targetIP>

-n

disables reverse DNS (avoids mixing DNS latency into path probe); -w wait, -q probes per hop; -I uses ICMP, -T -p uses TCP SYN to specified port. Implementations differ – run traceroute --help. TCP mode usually needs elevated privileges.

Seeing * at a hop but later hops appear means that router doesn't reply or rate-limits – not that business packets die there. Final hop missing may be because target doesn't reply to probes, while curl still succeeds. Paths can be asymmetric, ECMP, or MPLS; a single traceroute is just one observation.

Continuous monitoring suits mtr; report mode for evidence:

mtr --report --report-cycles 20 --no-dns <targetIP>
mtr --tcp --port <targetPort> --report --report-cycles 20 --no-dns <targetIP>

An intermediate hop showing high loss while later hops are clean usually indicates that node rate-limits probe packets, not that it drops forwarded traffic. Only when loss persists from a hop onward across all subsequent hops AND end-to-end business is affected does it constitute stronger evidence; also test from reverse direction or other sources. tracepath needs no special privileges and hints at path MTU – good supplement:

tracepath -n <targetIP>
tracepath6 -n <ipv6TargetIP> 2>/dev/null || tracepath -6 -n <ipv6TargetIP>

Tool name varies by distro; some only have tracepath with -6. Use local --help as reference.

Firewall: Examine Rules and Counters Before Changing

Modern distros may use nftables, or iptables as nftables frontend. Determine actual backend first; don't modify both simultaneously causing drift.

nft list ruleset
iptables --version
iptables-save
firewall-cmd --state 2>/dev/null || true
ufw status verbose 2>/dev/null || true

Rule output may contain internal addresses, ports, policies – sanitize before external tickets. Focus on input/output direction, interface, source/destination, state matching, default policy, and rule counters. nftables monitor trace observes packet hit path, but enabling trace modifies rules; in production limit to single source/destination/port, short duration, and delete diagnostic rules afterward. If organization has a firewall control plane, use its change process – no manual edits.

Example diagnostic flow (high-risk config change): save nft list ruleset, ensure remote management channel won't match, prepare rollback via console. meta nftrace set 1 marks matching packets for trace.

#!/usr/bin/env bash
set -euo pipefail

TARGET_IP="<targetIP>"
TARGET_PORT="<targetPort>"
BACKUP="/var/tmp/nft-before-$(date +%Y%m%d-%H%M%S).nft"

nft list ruleset > "${BACKUP}"
nft add table inet debug_trace
nft 'add chain inet debug_trace output { type filter hook output priority -301; policy accept; }'
nft add rule inet debug_trace output ip daddr "${TARGET_IP}" \
  tcp dport "${TARGET_PORT}" meta nftrace set 1

echo "Run in another terminal: nft monitor trace"
echo "Cleanup after capture: nft delete table inet debug_trace"
echo "Backup: ${BACKUP}"

Script covers IPv4 TCP only. After capture, explicitly clean up and verify business; delete the temporary debug_trace table, not production tables:

nft list table inet debug_trace
nft delete table inet debug_trace
nft list ruleset >/var/tmp/nft-after.txt

If script fails mid-way, manual cleanup still required. Don't restore with nft -f on backup unless semantics and atomic replace are confirmed; wrong restore can lock remote access. Prefer deleting only the explicitly created temporary table.

Server Return Path, Policy Routing, and rp_filter

TCP request reaches server but SYN-ACK doesn't return to client – common causes: wrong return route, missing policy routing, inconsistent SNAT, or reverse path filtering. On server, verify incoming packet and routing decision:

ip route get <clientIP>
ip rule show
sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.default.rp_filter
for f in /proc/sys/net/ipv4/conf/*/rp_filter; do
  printf '%s=%s
' "${f}" "$(cat "${f}")"
done

rp_filter=1

strict mode may drop legitimate packets in asymmetric routing or multi-homed scenarios; 2 loose mode; 0 disabled. Effective behavior also depends on per-interface values and kernel rules. Changing it alters security boundary – don't disable for “testing”. First prove reverse path check involvement via routing, captures, and kernel logs; then security review, per-interface canary, retain old value, verify return path.

conntrack and NAT

Connections through stateful firewall or NAT may hit conntrack table limits. Table full appears in kernel logs; NAT may exhaust source ports.

sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max 2>/dev/null || true
journalctl -k -g 'nf_conntrack.*table full' --since '-2 hours' --no-pager
conntrack -S 2>/dev/null || true
ss -s
sysctl net.ipv4.ip_local_port_range

Missing sysctl may mean module not loaded or not visible in current namespace – don't load modules arbitrarily in production. Raising table limit consumes memory; expanding ephemeral ports must avoid reserved ports. First identify root cause: abnormal short connections, pool leaks, long timeouts, attack traffic, or NAT gateway capacity.

Kubernetes: Test in the Correct Namespace and Network Location

All kubectl commands below explicitly use -n <namespace>. First check Pod IP, node, restarts, readiness; then Service, EndpointSlice, NetworkPolicy.

kubectl -n <namespace> get pod -o wide
kubectl -n <namespace> get service
kubectl -n <namespace> get endpointslice
kubectl -n <namespace> get networkpolicy
kubectl -n <namespace> describe pod <podName>

Service has ClusterIP but EndpointSlice lacks ready endpoints → issue in selector, readiness probe, or Pod status, not external link. NetworkPolicy effectiveness depends on CNI implementation; object existence alone isn't proof.

Execute inside the failing Pod for closest application path; requires image with tools and org permission for exec:

kubectl -n <namespace> exec <podName> -- getent ahosts <targetDomain>
kubectl -n <namespace> exec <podName> -- \
  curl --noproxy '*' -sS -o /dev/null \
  -w 'ip=%{remote_ip} code=%{http_code} total=%{time_total}
' \
  --connect-timeout 3 --max-time 10 https://<targetDomain>/health

If business image lacks troubleshooting tools, don't modify image or install packages in production. Use ephemeral debug containers per cluster policy; they create extra processes and traffic – confirm RBAC, audit, resource limits first.

kubectl -n <namespace> debug pod/<podName> \
  -it --image=<approvedDebugImage> --target=<containerName>

kubectl debug

parameters and ephemeral container capabilities depend on Kubernetes version, runtime, and security policies – run kubectl debug --help. <approvedDebugImage> must be a fixed version or digest from org registry; never pull unverified images ad-hoc.

Actionable Layered Workflows

Case 1: Domain Completely Unreachable

getent ahosts

fails → check NSS, resolv.conf, local resolved, DNS reachability, query return codes.

Resolution succeeds → for each returned A/AAAA run ip route get to confirm actual egress and source address.

ping IP fails → only means ICMP no reply; proceed to TCP port test.

telnet/nc connects → network layer and TCP handshake OK; move to TLS/HTTP.

curl with --resolve succeeds, normal domain fails → focus on DNS result or address family.

Case 2: ping Succeeds, Port Times Out

Client capture: confirm SYN sent.

Server capture: confirm SYN arrived.

SYN not at server → check client/middle firewall, security groups, routing, NAT.

SYN arrives but no SYN-ACK → check service listening, firewall, return route.

SYN-ACK sent but client doesn't see → check return path, security devices, asymmetric routing.

Case 3: Port Connects, curl Fails

curl -w

to determine failure in TLS or first byte.

TLS failure → check SNI, cert chain, time, protocol, cipher compatibility.

HTTP 4xx → check Host, path, auth, gateway policy.

HTTP 5xx → correlate reverse proxy and app logs with same timestamp, request ID, or trace ID.

Slow response → use phase timings to separate DNS, TCP, TLS, server processing.

Case 4: Only Some Machines Fail

Compare successful vs failing machines' configs instead of per-host guessing: addresses/masks, routing rules, DNS, proxy vars, time, CA certs, kernel params, firewall, cloud security groups, service mesh sidecar, egress NAT. Difference isn't root cause; verify with reproduction experiment.

Automated Collection: Read-Only, Time-Bounded, Auditable

Script below gathers four-tool results and local network state. No config changes; all commands timeout; single tool missing or failing won't stop collection. Root runs collect more process info; output may contain internal topology – sanitize before sharing.

#!/usr/bin/env bash
set -uo pipefail

TARGET_HOST="<targetDomain>"
TARGET_IP="<targetIP>"
TARGET_PORT="<targetPort>"
SCHEME="https"
OUT_DIR="/var/tmp/netcheck-$(date +%Y%m%d-%H%M%S)"

mkdir -p "${OUT_DIR}"
exec > >(tee -a "${OUT_DIR}/report.txt") 2>&1

run() {
  local name="$1"
  shift
  printf '
[%s] %s
' "$(date --iso-8601=seconds)" "${name}"
  timeout 20 "$@" || printf 'command_failed rc=%d
' "$?"
}

run identity uname -a
run links ip -br link
run addresses ip -br address
run rules ip rule show
run routes ip route show table all
run route_get ip route get "${TARGET_IP}"
run dns_getent getent ahosts "${TARGET_HOST}"
run ping ping -c 4 -W 2 "${TARGET_IP}"
run tcp nc -vz -w 3 "${TARGET_IP}" "${TARGET_PORT}"
run http curl --noproxy '*' -sS -o /dev/null \
  --connect-timeout 3 --max-time 10 \
  -w 'ip=%{remote_ip} code=%{http_code} dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total}
' \
  "${SCHEME}://${TARGET_HOST}:${TARGET_PORT}/health"
run trace traceroute -T -p "${TARGET_PORT}" -n -w 2 -q 1 "${TARGET_IP}"
run sockets ss -s

printf '
output_dir=%s
' "${OUT_DIR}"

Process substitution >(...) requires Bash. Script takes both domain and IP as explicit inputs to avoid DNS changes causing different commands to hit different targets. If nc or traceroute missing, script logs failure – don't install software during incident without approval.

Align Evidence with Timelines

Intermittent network issues fear unsynchronized clocks. Verify time sync first, then query logs on both ends within same time window. systemd-journald fields can output monotonic or UTC time for alignment.

timedatectl show -p NTPSynchronized -p TimeUSec -p Timezone
journalctl -u <serviceName> \
  --since '2026-08-05 10:00:00' \
  --until '2026-08-05 10:10:00' \
  -o short-iso-precise --no-pager

Replace example times with actual fault window. Client curl timestamps, load balancer access logs, service logs, and firewall events must align; otherwise causal ordering is unreliable.

Boundaries of High-Risk Fixes

Troubleshooting tools are mostly read-only; fixes can disrupt connectivity. Each action below requires separate impact review:

Modify default or policy routing – may immediately drop SSH. Save ip route / ip rule, verify out-of-band console, canary on single node; test temporary rules first, persistent config separate review.

Modify firewall/security groups – may expand exposure or block business. Verify source, destination, port, direction; use minimal-scope rules with counters; define explicit rollback steps.

Change MTU – affects all traffic on interface. Keep original value, test on drained redundant node, validate small/large packets, TLS, long-lived connections.

Restart network service – NetworkManager, systemd-networkd, or legacy network may rebuild all interfaces. Not a first step on remote production hosts.

Restart application – interrupts or migrates sessions. Confirm LB drain, readiness checks, connection draining, instance redundancy; roll one by one.

Flush conntrack – breaks existing stateful connections; rarely a routine fix. Address table-full root cause and capacity design first.

If routing change is unavoidable, protect with auto-rollback. Below is a conceptual flow; commands must adapt to distro network manager; ip runtime changes don't auto-persist to NetworkManager or systemd-networkd.

#!/usr/bin/env bash
set -euo pipefail

TARGET_CIDR="<targetCIDR>"
NEW_VIA="<newGatewayIP>"
IFACE="<interface>"
ROLLBACK_FILE="/var/tmp/routes-before-$(date +%Y%m%d-%H%M%S).txt"

ip route show table all > "${ROLLBACK_FILE}"
ip route get "${NEW_VIA}" >/dev/null

# Create 120s rollback task; change owner cancels transient unit after validation.
systemd-run --unit=route-auto-rollback --on-active=120 \
  /usr/sbin/ip route del "${TARGET_CIDR}" via "${NEW_VIA}" dev "${IFACE}"

ip route add "${TARGET_CIDR}" via "${NEW_VIA}" dev "${IFACE}"
ip route get "${TARGET_CIDR%/*}"
ping -c 3 -W 1 "${NEW_VIA}"

echo "Verify service now; cancel rollback only after validation:"
echo "systemctl stop route-auto-rollback.timer"
echo "route_snapshot=${ROLLBACK_FILE}"

Script assumes target CIDR had no prior same route, so rollback deletes new route; if replacing old route, this doesn't apply. TARGET_CIDR%/* extracts network address for generic CIDR but may not be a pingable host – only for route lookup demo. Before execution, write reversible commands based on actual old route, verify systemd transient timer created. Don't treat saved ip route show table all as executable restore script.

Common Misdiagnoses

“ping fails = server down”

Wrong. ICMP may be blocked by target, firewall, or cloud policy. Continue with TCP business port, server listening, and application health checks.

“telnet Connected = service healthy”

Wrong. Only proves TCP handshake. TLS may fail, HTTP may return 503, backend dependencies may be down. Continue with curl or protocol-specific client.

“traceroute stars from hop 7 = hop 7 broken”

Insufficient. Node may not return TTL exceeded; destination may still be reachable. Cross-validate with TCP traceroute, end-to-end curl, bidirectional tests, and ISP/cloud monitoring.

“curl -k works = cert fine”

Opposite: -k succeeds while normal verification fails means connection and app may be fine, but certificate trust or name validation has issues. Fix cert chain, SAN, CA, or system time – don't permanently disable verification.

“Connection refused = firewall drop”

Immediate refused usually means RST or active reject received – path at least partially reachable. Check target address, listening port, service status, firewall reject rules; silent drop more often shows as timeout.

“Disable firewall to test quickest”

Unacceptable. Global disable expands attack surface and destroys evidence of which rule matched. Use rule counters, limited captures, temporary precise trace, or single-source-single-port canary rules.

Evidence Checklist Before Delivering Root Cause

A troubleshooting conclusion must include:

Fault window, source, destination, explicit resolved IP and address family.

Local interface, addresses, ip route get output, actual source address.

ping conditions and results, clearly stating it only represents ICMP.

TCP test outcome (success, RST, timeout); if needed, dual-end packet captures.

curl remote IP, exit code, HTTP status, per-phase timings.

traceroute probe protocol used; don't equate non-responding hop with drop point.

Server listening address, process status, same-time-window logs, app health result.

Firewall/security-group/NetworkPolicy specific rules and counters – not “looks fine”.

If TLS involved: SNI, cert chain, validity, verification error, system time.

Pre/post fix validation under identical test conditions; confirm rollback still executable.

The correct use of the quartet is letting each command answer a limited question: ping checks ICMP basic reachability, telnet checks TCP handshake, curl checks DNS-to-application full flow, traceroute checks per-hop feedback. Then supplement with ip route get, ss, logs, rule counters, and controlled captures to converge “network unreachable” into a specific, fixable, verifiable, and rollback-able fault.