Understanding TCP: Protocol Basics, Handshakes, States, and Performance Optimizations

TCP is a connection‑oriented, reliable, byte‑stream transport‑layer protocol that guarantees ordered delivery and automatic discarding of duplicate segments.

TCP Header Format

The header includes fields such as sequence number, acknowledgment number, and control flags (ACK, RST, SYN, FIN). The sequence number is initialized with a random value during connection establishment and increments by the number of transmitted bytes.

Network Model

The article reviews the OSI seven‑layer model and the TCP/IP five‑layer model, showing their correspondence.

TCP Connection States

CLOSED – initial state

LISTEN – server socket waiting for connections

SYN_RCVD – SYN received

SYN_SENT – client has sent SYN

ESTABLISHED – connection established

TIME_WAIT – waiting for 2 MSL after FIN/ACK exchange

CLOSING – simultaneous FIN exchange

CLOSE_WAIT – waiting to close after receiving FIN

TIME_WAIT

TIME_WAIT exists to prevent old packets with the same four‑tuple from being accepted and to ensure the passive side receives the final ACK.

Excessive TIME_WAIT consumes memory and local port resources (32768‑61000 by default). The kernel parameter net.ipv4.ip_local_port_range can be inspected to view the port range. net.ipv4.ip_local_port_range Linux uses a fixed 2 MSL of 60 seconds (MSL = 30 seconds). The constant is defined in the kernel as:

#define TCP_TIMEWAIT_LEN (60HZ) /* about 60 seconds */

Modifying the TIME_WAIT duration requires recompiling the kernel.

Connection Process

Three‑Way Handshake

1st handshake: client sends SYN (seq = x), enters SYN_SENT.

2nd handshake: server replies SYN+ACK (seq = y, ack = x+1), enters SYN_RCVD.

3rd handshake: client sends ACK (ack = y+1); both sides enter ESTABLISHED. Data can be transferred after this step.

The third handshake may carry data; if it does, the next client segment uses the server’s ACK number as its sequence number.

Four‑Way Handshake (Connection Termination)

FIN from client → FIN_WAIT_1.

ACK from server → server enters CLOSE_WAIT, client enters FIN_WAIT_2.

FIN from server → server enters LAST_ACK, client enters TIME_WAIT.

ACK from client → both sides eventually reach CLOSED after waiting 2 MSL.

TCP Optimizations

Three‑Way Handshake Optimization

Four‑Way Handshake Optimization

Data Transfer Optimizations

TCP vs UDP

TCP is connection‑oriented, reliable, and provides flow and congestion control; UDP is connection‑less, best‑effort.

TCP header is 20 bytes (or more with options); UDP header is fixed 8 bytes.

TCP guarantees ordered, lossless delivery; UDP may drop or reorder packets.

TCP supports congestion control; UDP does not.

TCP is suitable for high‑accuracy, lower‑speed needs; UDP excels in low‑latency, real‑time scenarios.

Initial Sequence Number (ISN)

ISN is generated per RFC 1948 using a timer (increments every 4 ms) and a hash of the four‑tuple (source IP, source port, destination IP, destination port). The formula is:

ISN = M + F(localhost, localport, remotehost, remoteport)

where M is the timer value and F is a hash (MD5 is recommended).

Socket Workflow (TCP)

Server creates a socket and obtains a file descriptor.

Server binds the socket to an IP address and port.

Server calls listen to start listening.

Server calls accept to wait for a client connection.

Client calls connect to request a connection.

After the handshake, both sides have a connected socket.

Client writes data; server reads data.

When the client closes, the server receives EOF, processes remaining data, then closes.

Listen Backlog

The listen(socketfd, backlog) call defines the size of the accept queue (post‑kernel 2.2). The actual limit is the minimum of backlog and the kernel parameter somaxconn. The kernel maintains two queues: the incomplete (SYN) queue and the completed (accept) queue.

Additional Details

Linux kernel maintains the SYN queue for half‑opened connections and the accept queue for fully established connections. The client’s connect returns after the second handshake; the server’s accept returns after the third handshake.

When the client initiates close, it sends FIN, enters FIN_WAIT_1, and eventually reaches TIME_WAIT after the four‑way termination.

Overall, the article provides a comprehensive, step‑by‑step analysis of TCP’s mechanisms, state transitions, performance tuning parameters, and practical Linux commands for inspection and configuration.