Mastering TCP Handshake, Four‑Way Termination, and Common Faults

Real‑world scenario: timeouts, refusals and resets differ

When accessing an API you may see connection refused, connection reset by peer or a timeout. A timeout usually means the SYN or SYN‑ACK never arrived; a refusal means the target sent an RST because the port is closed or a policy rejected it; a reset indicates an active termination of the session. After a successful connection the first byte may be slow, which often points to the application layer rather than TCP.

Minimal checks :

getent ahosts api.example.internal
ip route get 10.20.30.40
sudo ss -ltnp '( sport = :443 )'
curl -vk --connect-timeout 5 https://api.example.internal/health

Replace the domain, IP and path with real values. The --connect-timeout flag limits only the connection phase.

Core principle: what the three‑way handshake confirms

The client sends a SYN with its initial sequence number; the server replies with SYN‑ACK, acknowledging the client’s sequence and providing its own; the client finishes with ACK. The states transition as follows:

Client                     Server
SYN, seq=x   --------->
            <----------- SYN‑ACK, seq=y, ack=x+1
ACK, ack=y+1 --------->
Both sides enter ESTABLISHED

Current socket states can be inspected with:

ss -tan state syn-sent
ss -tan state syn-recv
ss -tan state established '( sport = :443 or dport = :443 )'

Many SYN‑SENT sockets suggest DNS, routing, ACL or security‑group issues; many SYN‑RECV sockets indicate the server received SYNs but could not complete the final ACK, possibly due to SYN flood, listen‑queue pressure or return‑path problems.

Core principle: four‑way close and TIME‑WAIT

TCP is full‑duplex; each direction can be closed independently. The active closer sends FIN, the peer ACKs, may still send data, then sends its own FIN, and the active side ACKs and enters TIME‑WAIT to absorb lost ACKs and old packets.

Active closer          Passive side
FIN          --------->
            <----------- ACK
            <----------- FIN
ACK          --------->
Active side stays in TIME‑WAIT then closes

FIN and ACK can be combined, so captures may show fewer than four packets. TIME‑WAIT is normal for short‑lived connections, reverse proxies and high‑concurrency clients.

ss -tan state time-wait '( sport = :443 or dport = :443 )'
ss -s
cat /proc/net/sockstat

When TIME‑WAIT is abundant, determine whether it is on the client or server, whether connections are truly short‑lived, and avoid blindly shrinking kernel timeouts.

Verification experiment: double‑ended packet capture

Capture on the server for 30 seconds:

sudo timeout 30 tcpdump -ni any 'host 10.20.30.50 and tcp port 443' -nn -tttt -vv

Sample output shows a successful three‑way handshake but does not guarantee TLS or application success.

2026-07-10 14:00:00.100001 IP 10.20.30.50.52000 > 10.20.30.40.443: Flags [S], seq 1000, win 64240
2026-07-10 14:00:00.100120 IP 10.20.30.40.443 > 10.20.30.50.52000: Flags [S.], seq 2000, ack 1001, win 65160
2026-07-10 14:00:00.100300 IP 10.20.30.50.52000 > 10.20.30.40.443: Flags [.], ack 2001, win 502

Capture on the client side helps avoid single‑point mis‑interpretation.

Common fault 1: SYN timeout and path problems

Symptoms: curl hangs at the connect stage, client shows SYN‑SENT or repeated SYN retransmissions. Check in order: DNS resolution, routing, security‑group/ACL, server listening, whether the server receives the SYN, and whether the reply reaches the client.

sudo ss -ltnp '( sport = :443 )'
netstat -s | grep -Ei 'listen|SYN|retrans'
nstat -az | grep -E 'TcpExtListen|TcpRetransSegs|TcpTimeouts'

Metrics vary with kernel and iproute2 versions; compare increments over the same window rather than absolute counts.

Common fault 2: connection refused and RST

connection refused

means the target immediately returns RST. Verify the process is listening on the correct address and port:

sudo ss -ltnp '( sport = :443 )'
sudo lsof -nP -iTCP:443 -sTCP:LISTEN
systemctl status nginx --no-pager

Listening only on 127.0.0.1:443 will not accept remote connections; 0.0.0.0:443 covers IPv4, and [::]:443 covers IPv6. Do not blindly change to listen on all addresses without assessing exposure.

Common fault 3: fast handshake, slow first byte

After the handshake, TLS, HTTP, backend services, disk or CPU may delay the first byte. Use curl to split timings:

curl -sk -o /dev/null -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} first_byte=%{time_starttransfer} total=%{time_total}
' https://api.example.internal/health
ss -tin '( dport = :443 or sport = :443 )'

Example output:

dns=0.002 connect=0.004 tls=0.018 first_byte=1.442 total=1.443

. If TCP and TLS are fast but the first byte is slow, investigate the application or backend rather than tweaking TCP parameters.

Common fault 4: retransmission, window and MTU issues

Repeated segments or long periods without ACK may stem from packet loss, congestion, NIC errors, full queues, path MTU or slow receiver processing. Inspect the interface and socket:

ip -s link show dev eth0
ethtool -S eth0
ss -tin '( sport = :443 or dport = :443 )'

Test MTU without fragmentation (adjust size for IPv6 or tunnels):

ping -M do -s 1472 10.20.30.40

Changing MTU, routes or kernel TCP parameters affects all connections; back up configuration and test on a single node before rolling out.

Engineering recommendations

Standardise the investigation order: DNS → routing → listening → dual‑ended capture → socket state & counters → application segmentation. Evidence must show whether SYN arrived, SYN‑ACK returned, ACK completed, who sent FIN/RST, and whether the application slowed after TCP succeeded.

High‑connection services should minimise short connections: configure client connection pools and keep‑alive, align timeout policies across load balancers, proxies and backends, and monitor connection‑establish, TLS, first‑byte and error rates separately. Never adjust sysctl merely to “remove” TIME‑WAIT or SYN‑RECV; such changes can break port reuse and isolate old packets.

Troubleshooting runbook: minimal evidence set from client to server

Because network issues span teams, attach the following to every incident:

Timestamp with timezone

Client and server IP, port, protocol

Whether DNS, load balancer, proxy or VPN is involved

Exact client error message

Start times of captures on both ends

Server listening state

Recent configuration or deployment changes

Client‑side read‑only checks:

date -Is
getent ahosts api.example.internal
ip route get 10.20.30.40
ss -tan '( dport = :443 )'
curl -vk --connect-timeout 5 https://api.example.internal/health

Server‑side checks (replace <service_pid> with the real PID):

date -Is
sudo ss -ltnp '( sport = :443 )'
sudo ip addr show
sudo ip route show
sudo journalctl -u nginx --since '15 minutes ago' --no-pager

When the service is not managed by systemd, read its actual log files.

Why TCP state and application state can diverge

ESTABLISHED

only means the kernel believes the session exists; the application may still be blocked by thread‑pool exhaustion, backend latency, TLS errors or waiting for request bodies. Conversely, an application log showing a request start does not guarantee the client received a response—packet loss or premature client closure can break the round‑trip.

Therefore, distinguish four timestamps: DNS resolution, TCP connect, TLS handshake, and first application byte. For HTTPS, curl’s timing fields suffice; for custom protocols, record connect time and first protocol response time in client code.

Only when t_tcp_connected is noticeably slow should you revisit the three‑way handshake path; when t_first_byte - t_protocol_ready is slow, the problem likely lies in the application or downstream services.

Connection queue and file‑descriptor limits

Even if a process is listening, it may fail to accept new connections due to exhausted file descriptors or a full accept queue. Verify before increasing limits:

pidof nginx
cat /proc/<service_pid>/limits | grep -i 'open files'
sudo ls /proc/<service_pid>/fd | wc -l
sudo ss -ltnp '( sport = :443 )'

If the FD count is near the limit, the log shows “too many open files”, and new connections fail, then consider raising LimitNOFILE, adjusting the systemd unit or sysctl, but treat these as high‑risk changes with backups, gray‑scale rollout and verification.

Load balancer and connection timeout alignment

Four‑layer or seven‑layer load balancers introduce two TCP hops. A successful client handshake only proves the first hop; the backend connection must also be verified via ss, load‑balancer logs, health checks and source address inspection.

Timeouts must be aligned across layers: client connect timeout, load‑balancer idle timeout, reverse‑proxy read/write timeout, application request timeout, and downstream service timeout. Changing any timeout without evidence from logs or tracing can cause premature resets or 5xx errors.

Post‑mortem and prevention

Record whether the failure occurred during handshake, data transfer or termination; which side sent SYN/FIN/RST; capture timestamps; affected address families, subnets and load‑balancer paths; the fix applied and verification steps. For recurring issues, establish continuous monitoring of listen state, TCP retransmissions, NIC errors, connection‑queue depth, first‑byte latency and load‑balancer health, with clear owners for each metric.