Diagnosing Server Connectivity Issues with Ping, Telnet, Curl, and Traceroute

Define the fault and gather basics

Before any testing, record the five essential elements: source host (or pod), destination domain/IP, destination port, protocol, and time window. Verify whether the failure is universal or isolated, single‑port or all ports, and intermittent or persistent.

Capture the host’s identity, kernel version, and network namespace with commands such as:

date --iso-8601=seconds
hostnamectl
uname -r
systemd-detect-virt --container || true
readlink /proc/self/ns/net

If the problem originates inside a container, run all checks inside the same network namespace; a successful curl on the host only proves the host path, not the pod’s DNS, NetworkPolicy, service mesh, or SNAT.

Step 0 – Verify local interface and link state

Use ip -br link for a quick overview and ip -s link for error and drop counters. For physical NICs, check Link detected: no and inspect the cable, switch port, or bond members. For virtual NICs, examine ethtool output; rising RX dropped may indicate driver or queue issues, not external loss.

Step 1 – Let the kernel show the actual route

Do not rely on the default route alone. Run ip route get <targetIP> (or from <sourceIP>) to see which interface and next hop the kernel would use, taking policy routing, multiple tables, and VRF into account.

Step 2 – Verify layer‑2 neighbor resolution

For same‑subnet targets, check the ARP/NDP tables:

ip neigh show dev <iface>
ip neigh show to <gatewayIP> dev <iface>

If the gateway is reachable but the remote host is not, the failure is beyond layer 2. Avoid flushing the entire neighbor cache; clear only the offending entry after confirming impact.

Step 3 – DNS sanity check

Distinguish “DNS failed” from “ICMP failed” by first testing name resolution:

cat /etc/resolv.conf
getent ahosts <domain>
resolvectl query <domain> 2>/dev/null || true
dig +time=2 +tries=1 <domain> A
 dig +time=2 +tries=1 <domain> AAAA

If a specific DNS server is suspected, query it directly with dig @<dnsIP> <domain> A +noall +answer +comments or use dig +trace to see the full resolution chain.

Step 4 – ICMP reachability with ping

First ping the IP to rule out DNS, then ping the domain to see which address family is used. Use -c for count and -W for timeout; force IPv4 or IPv6 with -4 or -6 to detect dual‑stack issues. Example output:

64 bytes from 192.0.2.20: icmp_seq=1 ttl=56 time=12.4 ms
4 packets transmitted, 4 received, 0% packet loss

Interpret the results: a successful reply proves ICMP reachability, but packet loss or missing TTL does not guarantee application‑layer health.

Step 5 – TCP handshake with telnet (or nc )

telnet <IP> <port>

quickly shows whether a TCP three‑way handshake succeeds, times out, or is actively refused. Do not use telnet for UDP or to send credentials. For automation, nc -vz -w 3 <IP> <port> is preferred.

Step 6 – Full stack test with curl

curl -v --noproxy '*'

displays DNS lookup, TCP connect, TLS handshake, and HTTP response. Use timing flags to break down phases:

curl --noproxy '*' -sS -o /dev/null \
  --connect-timeout 3 --max-time 10 \
  -w 'remote_ip=%{remote_ip}
http_code=%{http_code}
namelookup=%{time_namelookup}
connect=%{time_connect}
appconnect=%{time_appconnect}
starttransfer=%{time_starttransfer}
total=%{time_total}
' \
  https://<domain>:<port>/health

Analyze the timing differences: time_connect - time_namelookup ≈ TCP handshake, time_appconnect - time_connect ≈ TLS, time_starttransfer - time_appconnect ≈ server processing. Use --resolve to force a specific IP while preserving the Host header and SNI.

Step 7 – Path tracing with traceroute

Run ICMP, UDP, or TCP‑SYN traceroute to see hop‑by‑hop TTL feedback. Example:

traceroute -n -w 2 -q 1 <IP>
traceroute -I -n -w 2 -q 1 <IP>
traceroute -T -p <port> -n -w 2 -q 1 <IP>

Asterisks (*) only mean the router did not reply; later hops may still be reachable. For continuous monitoring, use mtr --report --report-cycles 20 --no-dns <IP>. tracepath can also reveal MTU.

Step 8 – Firewall and nftables

Identify the actual firewall backend (nftables, iptables, firewalld, ufw) and list rules with nft list ruleset, iptables-save, or firewall-cmd --state. Examine counters to see which rules match. For high‑risk debugging, add a temporary trace rule:

nft add table inet debug_trace
nft add chain inet debug_trace output { type filter hook output priority -301; policy accept; }
nft add rule inet debug_trace output ip daddr <targetIP> tcp dport <targetPort> meta nftrace set 1

Run nft monitor trace in another terminal, then remove the temporary table after capture.

Step 9 – Conntrack and NAT limits

Check conntrack table size and saturation:

sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
journalctl -k -g 'nf_conntrack.*table full' --since '-2 hours'
conntrack -S
ss -s

Do not clear the table in production; instead, increase limits or investigate short‑lived connections.

Step 10 – Kubernetes‑specific checks

Always specify the namespace with kubectl -n <ns>. Verify pod IPs, service ClusterIP, EndpointSlice readiness, and NetworkPolicy status. When possible, exec into the pod and run the same getent, curl, and ping commands to test the exact network path the application uses. If the pod image lacks tools, use kubectl debug with an approved debug image.

Step 11 – Automated evidence collection script

The following Bash script runs all the above checks with timeouts, logs output to a directory, and never modifies the system:

#!/usr/bin/env bash
set -uo pipefail
TARGET_HOST="<domain>"
TARGET_IP="<IP>"
TARGET_PORT="<port>"
OUT_DIR="/var/tmp/netcheck-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$OUT_DIR"
exec > >(tee -a "$OUT_DIR/report.txt") 2>&1
run(){
  printf "
[%s] %s
" "$(date --iso-8601=seconds)" "$1"
  timeout 20 "$@" || printf "command_failed rc=%d
" $?
}
run identity uname -a
run links ip -br link
run addresses ip -br address
run rules ip rule show
run routes ip route show table all
run route_get ip route get "$TARGET_IP"
run dns_getent getent ahosts "$TARGET_HOST"
run ping ping -c 4 -W 2 "$TARGET_IP"
run tcp nc -vz -w 3 "$TARGET_IP" "$TARGET_PORT"
run http curl --noproxy '*' -sS -o /dev/null \
  --connect-timeout 3 --max-time 10 \
  -w 'ip=%{remote_ip} code=%{http_code} dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total}
' \
  https://$TARGET_HOST:$TARGET_PORT/health
run trace traceroute -T -p "$TARGET_PORT" -n -w 2 -q 1 "$TARGET_IP"
run sockets ss -s
printf "
output_dir=%s
" "$OUT_DIR"

The script records timestamps, command outputs, and exit codes, providing a reproducible audit trail.

Step 12 – Evidence checklist before concluding

Fault window, source, destination, resolved IPs and address families.

Interface, source address, and ip route get output.

Ping results (ICMP only).

TCP test outcome (success, RST, timeout) with optional packet captures.

Curl exit code, remote IP, HTTP status, and per‑phase timings.

Traceroute protocol and hop analysis.

Service listening sockets, systemd status, and logs.

Firewall/NAT/NetworkPolicy rules and counters.

TLS details (SNI, certificate chain, validity, system time).

Verification that the same test conditions succeed after any fix and that rollback steps are documented.

Common mis‑interpretations

“Ping fails → server down” – ICMP may be blocked; TCP may still work.

“Telnet Connected → service healthy” – only TCP handshake succeeded; TLS or HTTP may still fail.

“Traceroute star at hop 7 → hop 7 broken” – the router may simply not reply.

“curl -k succeeds → cert OK” – disabling verification hides real TLS problems.

“Connection refused = firewall drop” – Refused usually means RST from the host, not a silent drop.

High‑risk remediation boundaries

Changing default or policy routes – must snapshot ip route and ip rule, use a back‑channel, and apply changes on a single host first.

Modifying firewall or security‑group rules – use precise counters, temporary trace rules, and retain original configuration.

Adjusting MTU – test both small and large packets, verify TLS paths, and roll back if needed.

Restarting network services or the host – can break remote sessions; avoid as first step.

Clearing conntrack – disrupts all existing connections; only after confirming table exhaustion.

For any route change, the article provides a safe rollback script that creates a transient systemd timer to delete the new route after 120 seconds unless the operator cancels it.