Mastering TCP Handshakes, Teardowns, and Common Failures: A Practical Troubleshooting Guide

Real-World Scenario: Timeout, Refused, and Reset Are Not the Same

When users access an API, they may see timeout, connection refused, or connection reset by peer. Timeout usually means SYN never arrived, SYN-ACK never returned, or subsequent packets were dropped. Refused typically means the target responded with RST because the port is not listening or a policy explicitly rejects. Reset indicates one side actively terminated the session. Once the connection is established, slow first byte is rarely a TCP issue.

Start with minimal checks:

getent ahosts api.example.internal
ip route get 10.20.30.40
sudo ss -ltnp '( sport = :443 )'
curl -vk --connect-timeout 5 https://api.example.internal/health

Replace domain, IP, and health path with real service. --connect-timeout only limits connection establishment, not the entire request.

Core Principle: What the Three-Way Handshake Confirms

Client sends SYN with its initial sequence number; server replies SYN-ACK acknowledging client's sequence and providing its own; client sends final ACK. The three steps confirm both sides can send and receive, and synchronize sequence number spaces. Client after first SYN is in SYN-SENT; server after sending SYN-ACK is in SYN-RECV; after final ACK both enter ESTABLISHED.

Client                          Server
SYN, seq=x --------------->
                <----------- SYN-ACK, seq=y, ack=x+1
ACK, ack=y+1 --------------->
Both enter ESTABLISHED

View current states:

ss -tan state syn-sent
ss -tan state syn-recv
ss -tan state established '( sport = :443 or dport = :443 )'

Many client SYN-SENT entries: check DNS, egress routing, ACLs, security groups, target listening. Many server SYN-RECV: server received SYN but final ACK missing; verify return path, client policies, SYN flood, listen queue pressure. State is a snapshot; never conclude without packet captures and connection rates.

Listen queue is influenced by application listen(backlog), kernel net.core.somaxconn, net.ipv4.tcp_max_syn_backlog, and implementation varies by kernel version. Don't just increase one sysctl and declare optimization done; first confirm accept latency, SYN-RECV, application threads, and system resources jointly indicate queue bottleneck.

Core Principle: Four-Way Teardown and TIME-WAIT

TCP is full-duplex; each direction closes independently. Active closer sends FIN; peer ACKs, may still send data; peer then sends its FIN; active closer ACKs and enters TIME-WAIT to handle lost final ACK and stray packets.

Active Closer                  Passive Closer
FIN --------------->
                <----------- ACK
                <----------- FIN
ACK --------------->
Active side goes through TIME-WAIT then closes

ACK and FIN can be merged, so captures may not show four separate packets. TIME-WAIT is not a fault; short-lived clients, reverse proxies, and high concurrency all generate it:

ss -tan state time-wait '( sport = :443 or dport = :443 )'
ss -s
cat /proc/net/sockstat

When TIME-WAIT is high, first determine if it's on client or server, whether connections are truly short-lived, if client lacks connection reuse, or if upstream is retrying abnormally. Don't "clean TIME-WAIT", arbitrarily shorten kernel timeouts, or enable unknown sysctls. Wrong tuning breaks port reuse and old-packet isolation, creating harder-to-reproduce intermittent faults.

Verification Experiment: Dual-End Packet Capture to Confirm Handshake

On server, capture for 30 seconds against an authorized test client:

sudo timeout 30 tcpdump -ni any 'host 10.20.30.50 and tcp port 443' -nn -tttt -vv

Example output:

2026-07-10 14:00:00.100001 IP 10.20.30.50.52000 > 10.20.30.40.443: Flags [S], seq 1000, win 64240
2026-07-10 14:00:00.100120 IP 10.20.30.40.443 > 10.20.30.50.52000: Flags [S.], seq 2000, ack 1001, win 65160
2026-07-10 14:00:00.100300 IP 10.20.30.50.52000 > 10.20.30.40.443: Flags [.], ack 2001, win 502

Three complete packets show server observed successful handshake, but doesn't prove TLS or application are healthy. Server repeatedly sees same SYN without final ACK: check return path. Client repeats SYN, server has no record: check client egress, load balancer, ACLs, target address. Server sends RST: check listening, L4 proxy, or local policy.

Client-side simultaneous capture avoids single-point observation bias:

sudo timeout 30 tcpdump -ni any 'host 10.20.30.40 and tcp port 443' -nn -tttt -vv

Capture must limit host, port, and duration. Packets may contain addresses, protocol metadata, or unencrypted payloads; don't capture full production traffic long-term, and never share unsanitized pcaps.

Common Failure 1: SYN Timeout and Path Issues

Symptom: curl times out at connect phase, client shows SYN-SENT, capture shows SYN retransmissions. Check order: DNS resolves to correct address; routing correct; target security groups and ACLs; server listening; server received SYN; server reply reaches client.

Server check listening and kernel stats:

sudo ss -ltnp '( sport = :443 )'
netstat -s | grep -Ei 'listen|SYN|retrans'
nstat -az | grep -E 'TcpExtListen|TcpRetransSegs|TcpTimeouts'

netstat

, nstat fields vary by kernel and iproute2 version; use actual output. A single counter increase doesn't equal this failure; must compare delta in same observation window, packet captures, and traffic. With Prometheus, observe TCP retransmits, listen overflow, connection states, NIC errors; exact metric names depend on exporter.

Firewall changes are high-risk, affecting entire business port and management plane. Before executing: export existing rules, keep console access, precisely confirm source/destination/protocol/port, canary on smallest segment. Read-only first: sudo nft list ruleset Don't run flush ruleset, stop firewall service, or open 0.0.0.0/0 directly. Backup means saving ruleset and config management version; verify by connecting from authorized source and confirming management plane unaffected; rollback means restoring backed-up rules or declarative config previous version.

Common Failure 2: Connection Refused and RST

connection refused

usually means target immediately returns RST. Confirm service process listens on correct address and port:

sudo ss -ltnp '( sport = :443 )'
sudo lsof -nP -iTCP:443 -sTCP:LISTEN
systemctl status nginx --no-pager

lsof

may not be installed; fall back to ss. Listening on 127.0.0.1:443 won't accept remote connections; 0.0.0.0:443 covers IPv4, IPv6 needs [::]:443 and dual-stack policy. Don't blindly bind all addresses for reachability; must assess exposure, reverse proxy, and firewall.

RST after established: observe RST source address and timing. Real server sends RST: check process restart, application timeout, connection management, or proxy. Load balancer address sends RST: check its idle timeout, backend health, connection reuse. If from middle security device, network team must provide device evidence by 5-tuple and time window. Client's single reset error insufficient to assign blame.

Common Failure 3: Handshake Fast, First Byte Slow

After handshake, TLS, HTTP, backend dependencies, disk, CPU can all delay request. Use curl segmented timing, not lumping all latency into TCP:

curl -sk -o /dev/null -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} first_byte=%{time_starttransfer} total=%{time_total}
' https://api.example.internal/health
ss -tin '( dport = :443 or sport = :443 )'

Example output:

dns=0.002 connect=0.004 tls=0.018 first_byte=1.442 total=1.443

Example shows TCP and TLS fast, first byte slow; shift investigation to application and backend, not TCP parameters. Performance baseline must be sampled multiple times under same source, DNS, load window, and corroborated with application request logs, traces, or backend metrics.

Common Failure 4: Retransmissions, Window, and MTU

Duplicate segments, retransmissions, or long ACK gaps may stem from loss, congestion, NIC errors, queue full, path MTU, or slow receiver processing. Check interface and socket:

ip -s link show dev eth0
ethtool -S eth0
ss -tin '( sport = :443 or dport = :443 )'

Replace eth0 with real NIC. ethtool fields depend on driver; don't cross-machine interpret. Only large packets, VPN, or tunnel failures warrant non-fragment probe: ping -M do -s 1472 10.20.30.40 1472 + IPv4 28-byte header = 1500; IPv6 and tunnels adjust to actual MTU. If ICMP rate-limited, test inconclusive; don't change MTU based solely on this. Changing NIC MTU, routing, or kernel TCP params affects all host connections: before change, check dependent services and current network MTU, backup network config, canary on single node, verify bidirectional traffic and business requests, prepare console rollback.

Engineering Recommendations

Solidify troubleshooting order: name resolution, routing, listening, dual-end capture, socket states and counters, application segmentation. Root cause must have evidence: did SYN arrive? did SYN-ACK return? did ACK complete? who sent RST or FIN? did counters change in same time window? did application slow only after TCP success?

High-connection services should first reduce unnecessary short connections via engineering: properly configure client connection pools and keep-alive, align timeouts across load balancer, proxy, and backend, establish segmented monitoring for connect time, TLS time, first-byte time, and error rate. Don't blindly tune sysctl to eliminate TIME-WAIT or SYN-RECV symptoms. Every firewall change, kernel param change, service restart, load balancer switch, or scale operation must document impact scope, pre-checks, backup, canary or dry-run, verification, and rollback.

Three-way handshake and four-way teardown are not interview answers; they are a coordinate system for reading evidence. Put packet captures, ss, kernel logs, and application latency on the same timeline, and network faults transform from "feels slow" into verifiable, rollback-able engineering problems.

Troubleshooting Runbook: Minimum Evidence Set from Client to Server

Network issues often span multiple teams. To let network, platform, application, and security teams collaborate on same facts, each ticket should attach minimum evidence set: occurrence time (with timezone); client and server IP, port, protocol; whether DNS, load balancer, proxy, NAT, or VPN involved; client error verbatim; both-end capture start times; server listening state; recent config or release changes. Don't use "user cannot access" as sole info; it cannot locate which hop lost the packet.

Client executes read-only checks in order:

date -Is
getent ahosts api.example.internal
ip route get 10.20.30.40
ss -tan '( dport = :443 )'
curl -vk --connect-timeout 5 https://api.example.internal/health

Server records:

date -Is
sudo ss -ltnp '( sport = :443 )'
sudo ip addr show
sudo ip route show
sudo journalctl -u nginx --since '15 minutes ago' --no-pager

Here nginx is example service name; replace with actual systemd unit for listening process; if not systemd-managed, read its real log path. Before test, confirm client's actual post-NAT source address, because server capture filters on visible source IP, not necessarily user's private address.

Why TCP State and Application State May Contradict

ESTABLISHED

only means kernel thinks TCP session established; doesn't mean peer business thread is processing request. An application can accept connection then stall due to thread pool exhaustion, backend connection block, TLS negotiation error, or waiting to read request body. Conversely, application log showing request start doesn't guarantee client received response; return-path loss or client early close can make final response undeliverable.

Therefore troubleshooting must distinguish four timestamps: DNS returned address; TCP connect completed; TLS handshake completed; HTTP first byte or application protocol first response arrived. For HTTPS, use curl's timing fields; for custom TCP protocols, client can record connect success time and first protocol response time, or use application's structured logs. Pseudocode example (not directly executable):

record(t_dns_done)
connect(remote_address, timeout)
record(t_tcp_connected)
perform_protocol_handshake()
record(t_protocol_ready)
send_request(); read_first_response_byte()
record(t_first_byte)

Only when t_tcp_connected significantly slows, prioritize checking three-way handshake path; when t_first_byte - t_protocol_ready slows, usually hand off to application or dependency chain. This distinction avoids blaming kernel parameters.

Connection Queues and File Descriptors: When to Check

Service process may listen on port but still fail connections due to process file descriptor exhaustion, delayed accept, or application thread block. Get facts first, don't just increase limits:

pidof nginx
cat /proc/<service_pid>/limits | grep -i 'open files'
sudo ls /proc/<service_pid>/fd | wc -l
sudo ss -ltnp '( sport = :443 )'

<service_pid>

must be real service PID; first two commands need permission to read that process. When fd count nears limit, same window service logs show too many open files, and new connections fail — all three evidences together justify handling limit or leak. High fd count alone doesn't indicate error; high-concurrency proxies naturally hold many connections.

Raising LimitNOFILE, modifying systemd unit, restarting service, or changing sysctl are high-risk changes affecting service restart window, connection ceiling, and resource usage. Pre-check: existing unit source, application supported ceiling, host memory, redundant replicas; backup current unit override or config management version; canary on single replica; verify new connections, old connections, fd usage, error logs; rollback by restoring original override and restarting that replica per change window. Don't directly edit vendor unit under /usr/lib/systemd/system; upgrades may overwrite; follow local systemd override and config management standards.

Load Balancer and Connection Timeouts: Must Align Layers

Via L4 or L7 load balancer, at least two TCP connections exist: client-to-LB and LB-to-backend. Client successful handshake only proves first segment normal; backend ss normal doesn't prove LB selected that node. Troubleshoot by separately confirming VIP, LB logs, backend health, backend source address, and each segment's timeout config.

Timeout configs should align by layer: client connect timeout, LB idle timeout, reverse proxy read/write timeout, application request timeout, database/downstream timeout. No universal fixed seconds; if upper timeout shorter than normal business latency, it actively tears down still-working downstream requests; if lower timeout significantly shorter, it may repeatedly return resets or 5xx upstream. Before changing timeouts, extract percentile and max legitimate request evidence from access logs and traces, canary on small backend pool or single route, confirm connection count, memory, queueing not abnormally amplified.

LB traffic shift, backend removal, scale, and failover are high-risk. Impact includes connection disruption, session stickiness loss, cache warmup, data consistency. Pre-check: healthy replicas, session/state migration strategy, cutback conditions; backup: save current backend pool and routing config; canary: shift small traffic percentage or one reclaimable node; verify: connection establishment, key requests, error rate, latency all meet targets; rollback: gradually revert traffic per recorded old config, never repeatedly full-switch during anomaly.

Postmortem and Prevention: Make Packet Evidence Usable

Postmortem should record whether issue was in handshake, data transfer, or close phase; who sent SYN, SYN-ACK, ACK, FIN, or RST; time delta between dual-end captures and logs; affected address families, network segments, LB paths; final fix and its verification. For recurring connection issues, establish continuous observation instead of ad-hoc captures: service listening state, TCP retransmits, NIC errors, connection queues, application first-byte time, LB backend health, DNS resolution failure rate — all with clear owners.

Monitoring only alerts anomalies; cannot automatically replace root-cause evidence. Example: TCP retransmit rise could be path loss, server CPU saturation, NIC anomaly, or receiver application block; must return to same time window captures, system resources, application logs. Solidifying this evidence chain prevents repeated guessing in every connection fault.