How to Diagnose Unreachable Server Networks with Ping, Telnet, Curl, and Traceroute

Define the fault: source, destination, protocol, and time

Before any test, record the five key elements – source host or pod, target domain/IP, target port, transport protocol, and the failure window. Determine whether the problem affects all sources, a single host, all ports, or a specific service, and whether it is continuous or intermittent.

Step 0: Verify local configuration

Interface, address, and link status

Run ip -br link for a quick overview and ip -s link to see error and drop counters. Use ethtool <interface> to check link detection, speed, and duplex. A "Link detected: no" indicates a physical problem (cable, switch port, bond member, driver).

Neighbor resolution on the same subnet

Inspect the ARP (IPv4) or NDP (IPv6) tables with ip neigh show dev <interface>. A FAILED or INCOMPLETE entry means layer‑2 resolution has not completed. Verify the gateway first; if the gateway is reachable but the remote host is not, the issue is beyond the local link.

DNS must be split into resolution success and correctness

Use getent ahosts <domain> to follow the system's Name Service Switch configuration, then query specific DNS servers with dig +time=2 +tries=1 <domain> A and dig +time=2 +tries=1 <domain> AAAA. When systemd-resolved is in use, resolvectl query <domain> shows per‑interface DNS settings.

ping: verify ICMP reachability and quality

Separate IP and domain testing

First ping the raw IP to rule out DNS, then ping the domain to see which address family is chosen. Example:

ping -c 4 -W 2 <target_ip>
ping -4 -c 4 -W 2 <target_domain>
ping -6 -c 4 -W 2 <target_domain>

Note the -c (count) and -W (per‑reply timeout) semantics; BusyBox may differ, so check ping --help first.

Fix source interface or source address

On multi‑NIC hosts, force the source with ping -I <interface> -c 4 -W 2 <target_ip> or ping -I <source_ip> -c 4 -W 2 <target_ip>. Compare the results with a plain ping; mismatches often point to missing routes, policy routing, or upstream filtering.

MTU/PMTU black‑hole detection

Large packets may be dropped while small ones succeed. Test with DF set:

ping -4 -M do -s 1472 -c 3 -W 2 <target_ip>
ping -4 -M do -s 1400 -c 3 -W 2 <target_ip>

If 1400 succeeds but 1472 fails, halve the payload until the threshold is found, then investigate tunnels, VPNs, or ICMP "Fragmentation Needed" filtering. Do not change the interface MTU globally without a controlled test.

Correct interpretation of ping failures

Name or service not known

: DNS/NSS problem, not ICMP. Network is unreachable: No matching route; check address, routing table, and policy rules. Destination Host Unreachable: Message source (local, gateway, or router) indicates where the path broke.

All timeouts: Could be silent drop, ICMP blocked, or the remote host down – cannot be blamed on a single cause without more evidence.

High loss or latency: Use a longer, controlled sample and tools like mtr or interface counters; ICMP may be rate‑limited, so results do not always reflect TCP behavior.

telnet: only verifies TCP connection, not application health

While nc is better for automation, telnet <host> <port> quickly shows three states: Connected to …: TCP three‑way handshake succeeded. Use Ctrl+] then quit to exit. Connection refused: The port is closed or a firewall actively rejected it.

Long timeout: SYN or SYN‑ACK was dropped; capture packets to locate the loss.

Do not send credentials over telnet for production services.

curl: split DNS, TCP, TLS, and HTTP timing

Run with verbose output and no proxy to isolate the path:

curl --noproxy '*'
     -v \
     --connect-timeout 3 \
     --max-time 10 \
     https://<domain>:<port>/health

Output comparable stage timings

Use -w to get per‑phase durations and the remote IP actually used:

curl --noproxy '*'
     -sS -o /dev/null \
     --connect-timeout 3 --max-time 15 \
     -w 'remote_ip=%{remote_ip}
http_code=%{http_code}
namelookup=%{time_namelookup}
connect=%{time_connect}
appconnect=%{time_appconnect}
starttransfer=%{time_starttransfer}
total=%{time_total}
' \
     https://<domain>:<port>/health

Differences such as time_connect - time_namelookup approximate the TCP handshake time, while time_appconnect - time_connect reflects TLS setup.

Bypass DNS while preserving Host and SNI

Force the IP but keep the original hostname with --resolve:

curl --noproxy '*'
     -v \
     --resolve '<domain>:<port>:<ip>'
     --connect-timeout 3 --max-time 10 \
     https://<domain>:<port>/health

If the normal request fails but the --resolve version succeeds, the problem lies in DNS resolution, caching, or address‑family selection.

Separate IPv4 and IPv6

Dual‑stack domains may have one broken address family. Test both explicitly:

curl -4 --noproxy '*'
     -sS -o /dev/null \
     -w 'ip=%{remote_ip} code=%{http_code} total=%{time_total}
' \
     --connect-timeout 3 --max-time 10 https://<domain>/

curl -6 --noproxy '*'
     -sS -o /dev/null \
     -w 'ip=%{remote_ip} code=%{http_code} total=%{time_total}
' \
     --connect-timeout 3 --max-time 10 https://<domain>/

If only IPv6 fails, inspect the IPv6 address, default route, neighbor discovery, and firewall rules before disabling IPv6 globally.

TLS failures must check certificate chain, name, and time

A certificate verify failed error can stem from expired certificates, wrong system time, missing intermediate CAs, SNI mismatch, or a broken local CA store. Do not simply add -k; instead, examine the certificate with OpenSSL:

date -u
timedatectl status
openssl s_client -connect <domain>:<port> \
    -servername <domain> -verify_return_error

Extract key fields with:

openssl s_client -connect <domain>:<port> -servername <domain> -showcerts

and pipe to

openssl x509 -noout -subject -issuer -dates -ext subjectAltName

if the OpenSSL version supports -ext.

HTTP status is not a network‑layer failure

Successful 200 means end‑to‑end success. Redirects ( 301/302), authentication codes ( 401/403), not‑found ( 404), or upstream errors ( 502/503/504) indicate application or proxy issues. Use -D to capture headers and limit the body with --range 0-4095 for safe inspection.

Proxy variables often cause per‑user failures

Check environment variables in the host, systemd service, and container:

env | grep -iE '^(http|https|all|no)_proxy='
systemctl show <service> -p Environment
curl -v --connect-timeout 3 https://<domain>/ 2>&1 | grep -E 'Uses proxy env variable|Connected to|Trying '

Compare a run with --noproxy '*' to isolate proxy‑related DNS, ACL, or authentication problems.

traceroute: locate path boundaries, don’t treat * as a break

Classic traceroute sends increasing TTL packets. Use the appropriate probe type for the service:

traceroute -n -w 2 -q 1 <target_ip>          # default (UDP on many Linuxes)
traceroute -I -n -w 2 -q 1 <target_ip>      # ICMP
traceroute -T -p <port> -n -w 2 -q 1 <target_ip>  # TCP SYN

A * on a hop only means that router did not reply; the path may still be functional. Use mtr --report --report-cycles 20 --no-dns <target_ip> for continuous observation and tracepath -n <target_ip> to see the path MTU.

Firewall: inspect rules and counters before changing

Modern systems may use nftables with an iptables front‑end. List the active ruleset with nft list ruleset, then check counters with iptables-save or ufw status verbose. Focus on direction, interface, source/destination, and match counts. Use nft monitor trace only on a temporary table and remove it after the test.

conntrack and NAT

Stateful firewalls and NAT rely on the conntrack table. Verify size and usage with:

sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
journalctl -k -g 'nf_conntrack.*table full' --since '-2 hours' --no-pager
conntrack -S
ss -s
sysctl net.ipv4.ip_local_port_range

Do not clear the table in production; instead, identify short‑lived connections, connection‑pool exhaustion, or capacity limits.

Kubernetes scenario: test in the correct namespace and network location

All kubectl commands must specify -n <namespace>. Verify pod IP, node, readiness, Service, EndpointSlice, and NetworkPolicy before testing from inside the pod:

kubectl -n <ns> get pod -o wide
kubectl -n <ns> get service
kubectl -n <ns> get endpointslice
kubectl -n <ns> get networkpolicy
kubectl -n <ns> exec <pod> -- getent ahosts <domain>
kubectl -n <ns> exec <pod> -- curl --noproxy '*' -sS -o /dev/null \
    -w 'ip=%{remote_ip} code=%{http_code} total=%{time_total}
' \
    --connect-timeout 3 --max-time 10 https://<domain>/health

If the image lacks debugging tools, use

kubectl debug pod/<pod> -it --image=<approved_debug_image> --target=<container>

(available on newer clusters) to create a temporary container with the needed utilities.

A complete layered procedure

Case 1: Domain completely unreachable

Run getent ahosts <domain>. Failure points to NSS, /etc/resolv.conf, or local systemd-resolved.

If resolution succeeds, for each returned IP run ip route get <ip> to confirm the egress interface and source address.

If ping <ip> fails, continue with TCP port testing.

If telnet/nc connects, move to TLS/HTTP checks.

Use curl --resolve to verify whether DNS is the culprit.

Case 2: Ping succeeds, port times out

Capture client packets to confirm SYN is sent.

Capture server packets to confirm SYN arrival.

If SYN never reaches the server, investigate client/network firewalls, security groups, routing, or NAT.

If SYN reaches but no SYN‑ACK, check service listening, firewall, and return path.

If SYN‑ACK is sent but not seen by the client, examine return routing, asymmetric paths, or reverse‑path filtering.

Case 3: Port connects, curl fails

Use curl -w to see whether the failure occurs in TLS or after the request.

For TLS failures, verify SNI, certificate chain, system time, and cipher compatibility.

For HTTP 4xx, check Host header, path, and authentication.

For HTTP 5xx, correlate with reverse‑proxy or backend logs using request IDs.

For slow responses, compare the stage timings to isolate DNS, TCP, TLS, or application processing delays.

Case 4: Only some machines fail

Compare configuration differences (IP/mask, routes, DNS, proxy variables, time, CA bundle, kernel parameters, firewall, cloud security groups, service‑mesh sidecar, source NAT). Validate each hypothesis with a controlled experiment before concluding.

Automated collection: read‑only, timed, auditable

The following script gathers the four‑tool results and local network state without modifying anything. It logs timestamps, runs each command with a timeout, and writes a summary directory.

#!/usr/bin/env bash
set -uo pipefail

TARGET_HOST="<domain>"
TARGET_IP="<ip>"
TARGET_PORT="<port>"
SCHEME="https"
OUT_DIR="/var/tmp/netcheck-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$OUT_DIR"
exec > >(tee -a "$OUT_DIR/report.txt") 2>&1

run() {
  local name="$1"; shift
  printf "
[%s] %s
" "$(date --iso-8601=seconds)" "$name"
  timeout 20 "$@" || printf "command_failed rc=%d
" $?
}

run "identity" uname -a
run "links" ip -br link
run "addresses" ip -br address
run "rules" ip rule show
run "routes" ip route show table all
run "route_get" ip route get "$TARGET_IP"
run "dns_getent" getent ahosts "$TARGET_HOST"
run "ping" ping -c 4 -W 2 "$TARGET_IP"
run "tcp" nc -vz -w 3 "$TARGET_IP" "$TARGET_PORT"
run "http" curl --noproxy '*' -sS -o /dev/null \
    --connect-timeout 3 --max-time 10 \
    -w 'ip=%{remote_ip} code=%{http_code} dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total}
' \
    "$SCHEME://$TARGET_HOST:$TARGET_PORT/health"
run "trace" traceroute -T -p "$TARGET_PORT" -n -w 2 -q 1 "$TARGET_IP"
run "sockets" ss -s
printf "
output_dir=%s
" "$OUT_DIR"

Align timestamps on both sides

Ensure NTP synchronization, then query logs with matching windows. Example:

timedatectl show -p NTPSynchronized -p TimeUSec -p Timezone
journalctl -u <service> --since '2026-08-05 10:00:00' --until '2026-08-05 10:10:00' -o short-iso-precise --no-pager

Correlate client curl timestamps, load‑balancer logs, service logs, and firewall events to establish causality.

Boundaries of high‑risk fixes

Changing default or policy routes can drop SSH; back up ip route and ip rule before editing, test on a single host, and keep a rollback plan.

Modifying firewall or security‑group rules may expose services; use precise rules with counters and a clear undo step.

Adjusting MTU affects all traffic on the interface; test with small and large packets, TLS handshakes, and keep the original value for rollback.

Restarting network managers (NetworkManager, systemd‑networkd) rebuilds interfaces and can disconnect remote sessions; avoid as a first step.

Restarting applications interrupts sessions; ensure load‑balancer drain, readiness probes, and graceful shutdown.

Flushing conntrack breaks existing connections; address table‑full causes before clearing.

When a route change is unavoidable, create a systemd transient timer that automatically reverts after a safe interval, and keep a snapshot of the original routing table.

Common mis‑judgments

"Ping failure means the server is down" – ICMP may be blocked; continue with TCP checks.

"Telnet Connected means the service is healthy" – only TCP handshake succeeded; TLS or HTTP may still fail.

"Traceroute * at hop 7 means that hop is broken" – the router simply does not reply; the path may still be functional.

"curl -k success proves the certificate is fine" – disabling verification hides the real problem; fix the certificate chain instead.

"Connection refused equals firewall drop" – it usually means the port is closed or actively rejected.

"Turn off the firewall to test" – disables security and removes evidence; use rule counters or temporary trace rules.

Evidence checklist before delivering root cause

Failure window, source, target, and resolved IPs.

Local interface, address, and ip route get output.

Ping parameters and results (ICMP only).

TCP test outcome: success, RST, or timeout, with optional packet captures.

Curl exit code, remote IP, HTTP status, and per‑stage timings.

Traceroute protocol used and hop interpretation.

Service listening address, process state, and relevant logs.

Firewall/ security‑group / NetworkPolicy rules and counters.

If TLS is involved, record SNI, certificate chain, validity dates, and verification error.

Validate the fix with the same test conditions and keep a rollback plan.

Proper use of the four‑tool set

ping

– basic ICMP reachability. telnet – TCP connection verification. curl – end‑to‑end DNS, TCP, TLS, and HTTP timing. traceroute – hop‑by‑hop path feedback.

Supplement with ip route get, ss, logs, rule counters, and controlled captures to narrow the fault to a reproducible, fixable, and rollback‑able cause.