Mastering TCP States: Diagnose Network Issues with Linux Tools

1. TCP States

Understanding TCP states helps locate network or system faults.

Common Linux commands to view TCP status:

netstat -nat – shows the count of each TCP state.

lsof -i:port – displays open sockets for a given port.

sar -n SOCK – shows the number of TCP connections created.

tcpdump -iany tcp port 9000 – captures packets on port 9000.

Typical network testing commands:

ping – tests latency, jitter, and packet loss; often disabled on servers for security.

traceroute hostname – traces the route to a host.

pathping www.baidu.com – combines ping and traceroute.

mtr – merges ping, nslookup, and traceroute features.

nslookup – resolves domain names and checks DNS configuration.

Key TCP states:

LISTENING – server is waiting for incoming connections; a service must be running for the port to be in this state.

SYN‑SENT – client has sent a SYN and is awaiting a matching response.

SYN‑RECEIVED – server has received a SYN and sent SYN‑ACK.

ESTABLISHED – data can be transferred between client and server.

FIN‑WAIT‑1 , FIN‑WAIT‑2 , TIME‑WAIT , CLOSE‑WAIT , CLOSING , LAST‑ACK , CLOSED – various stages of connection termination.

The socket is actively attempting to establish a connection.

When many SYN‑SENT entries appear, possible causes include unreachable hosts, port scans, or malware generating SYN floods.

A large number of SYN‑RECEIVED entries often indicates a SYN‑Flood DoS attack, where forged SYN packets exhaust server resources.

During normal termination, the client moves through FIN‑WAIT‑1 → FIN‑WAIT‑2 → TIME‑WAIT → CLOSED, while the server moves through CLOSE‑WAIT → LAST‑ACK → CLOSED.

Connection is closed, and the socket is waiting for a shutdown from the remote end.

2. TCP State Transition Diagram

The diagram shows separate client and server state machines. Client path: CLOSED → SYN‑SENT → ESTABLISHED → FIN‑WAIT‑1 → FIN‑WAIT‑2 → TIME‑WAIT → CLOSED. Server path: CLOSED → LISTEN → SYN‑RECEIVED → ESTABLISHED → CLOSE‑WAIT → LAST‑ACK → CLOSED.

3. TCP Three‑Way Handshake

To establish a connection:

Client sends SYN (active open) → socket enters SYN‑SENT.

Server replies with SYN‑ACK → server enters SYN‑RECEIVED.

Client sends ACK → both sides enter ESTABLISHED.

Example with netstat -nat and tcpdump shows the three packets and sequence numbers.

4. TCP Four‑Way Termination

Because TCP is full‑duplex, each direction must be closed separately:

Client sends FIN → enters FIN‑WAIT‑1.

Server ACKs the FIN → server enters CLOSE‑WAIT.

Server sends its own FIN → enters LAST‑ACK.

Client ACKs the server’s FIN → enters TIME‑WAIT, then CLOSED.

The TIME‑WAIT state (2 MSL) ensures delayed packets are discarded and the final ACK can be retransmitted if lost.

5. Simultaneous Open

Both ends may issue an active open, each sending a SYN. The connection is established after four packet exchanges, though most BSD‑derived TCP/IP stacks do not support this.

6. Simultaneous Close

If both sides send FIN at the same time, they move to CLOSING and then to TIME‑WAIT after exchanging ACKs, requiring four packets just like the normal termination.

7. TCP Flags

Important flag bits:

SYN – initiates a connection.

FIN – terminates a connection.

ACK – acknowledges received data.

PSH – indicates data payload.

RST – resets the connection.

SYN and FIN are never set together; ACK can accompany SYN or FIN. RST is usually seen after a FIN when the connection is being reset.

8. Handling Unexpected Client Disconnects

If a client crashes without closing the socket, the server may block on recv or send. Use keep‑alive mechanisms to detect half‑open connections.

Two approaches:

Application‑level heartbeat packets.

Enable TCP keep‑alive (system‑wide or per‑socket) with tuned parameters.

Typical keep‑alive settings (in /etc/sysctl.conf) are:

net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 20
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30

Programmatically, set SO_KEEPALIVE and adjust the interval, probes, and timeout as needed.

9. Common Linux Socket Error Codes (errno)

22 – Invalid argument (e.g., malformed IP address).

101 – Network unreachable.

111 – Connection refused.

115 – Operation now in progress (non‑blocking socket without immediate response).