Understanding TCP: OSI Layers, Header Fields, Handshakes and Common Issues

Introduction

Hello, I’m Panpan. After writing about algorithms and Linux commands, I now explore network protocols, which are essential knowledge for both client‑side and server‑side development.

What Is TCP?

TCP (Transmission Control Protocol) is a connection‑oriented, reliable, byte‑stream transport‑layer protocol. Its main characteristics are:

Connection‑oriented : communication occurs between a single pair of endpoints.

Reliable delivery : TCP guarantees that a segment reaches the receiver despite network changes.

Byte‑stream : data is treated as a continuous stream of bytes.

Network Models

Seven‑Layer OSI Model

The ISO‑defined OSI model consists of seven layers, each with typical devices and data units. Data is encapsulated layer by layer as shown in the diagram.

When a host sends user data, the process starts at the application layer and proceeds downward, adding headers at each layer (Application Header AH, Presentation Header PH, Session Header SH, Transport Header TH, Network Header NH, Data‑link Header DH, and finally the frame trailer DT).

Five‑Layer Model

Application Layer : defines protocols such as HTTP, SMTP, etc.

Transport Layer : provides TCP (connection‑oriented) and UDP (connectionless) services.

Network Layer : handles packet routing and IP addressing.

Data‑link Layer : assembles IP packets into frames.

Physical Layer : transmits the bit stream.

Four‑Layer TCP/IP Model

The most widely used model for practical networking consists of:

Network Interface Layer : implements protocols like ARP that bridge IP data to the physical medium.

Internet Layer : includes IP, RIP, and ICMP for routing and diagnostics.

Transport Layer : provides TCP (reliable) and UDP (unreliable) services.

Application Layer : merges the OSI application and presentation layers; common protocols are HTTP, FTP, SMTP, etc.

Thus, TCP operates at the fourth OSI layer (Transport), IP at the third (Network), and ARP at the second (Data‑link). Data is first packaged into a TCP segment, then an IP packet, and finally an Ethernet frame.

TCP Header Format

The TCP header contains several fields that manage connections and control data flow:

16‑bit Source and Destination Ports : identify the sending and receiving processes.

32‑bit Sequence Number : numbers each byte in the byte stream.

32‑bit Acknowledgement Number : confirms receipt of data (seq + 1).

4‑bit Header Length : indicates the size of the TCP header (max 60 bytes).

6‑bit Flags : URG, ACK, PSH, RST, SYN, FIN, each controlling specific connection states.

16‑bit Window Size : the receiver’s advertised buffer space for flow control.

16‑bit Checksum : validates the integrity of the header and payload.

16‑bit Urgent Pointer : points to urgent data within the stream.

A TCP connection is identified by a four‑tuple (src_ip, src_port, dst_ip, dst_port) – often called a five‑tuple when the protocol field is included.

Three‑Way Handshake

TCP establishes a connection through three steps:

SYN : client sends a SYN segment and enters SYN_SENT.

SYN‑ACK : server acknowledges with SYN‑ACK and enters SYN_RECEIVED.

ACK : client replies with ACK, and both sides move to ESTABLISHED.

After the handshake, data transmission can begin.

Handshake Anomalies

If the server does not receive the client’s ACK, it will retransmit SYN‑ACK up to five times (intervals 1 s, 2 s, 4 s, 8 s, 16 s). After the fifth timeout, the connection is dropped.

SYN‑Flood attacks exploit this by sending many spoofed SYNs, filling the server’s SYN‑RECEIVED queue. Mitigations include enabling tcp_syncookies = 1, adjusting netdev_max_backlog, or setting tcp_abort_on_overflow to reject excess SYNs.

Four‑Way Termination

Closing a TCP connection involves four steps (often described as a “four‑way handshake”):

Client sends FIN, entering FIN_WAIT_1.

Server acknowledges with ACK, moving client to FIN_WAIT_2.

Server sends its own FIN, entering CLOSE_WAIT on the client.

Client acknowledges with ACK, entering TIME_WAIT before finally moving to CLOSED.

The process ensures both directions are gracefully shut down.

Termination Anomalies

Excessive TIME_WAIT sockets can exhaust memory and port resources under high concurrency. Common mitigations:

Enable tcp_tw_reuse = 1 (client‑side only) to reuse eligible TIME_WAIT sockets.

Enable tcp_timestamps = 1 to embed timestamps and avoid the 2 MSL wait.

Adjust tcp_max_tw_buckets (default 18000) to limit the number of TIME_WAIT entries, though this is a blunt tool.

TCP State Diagram

The TCP state machine includes states such as CLOSED, LISTEN, SYN_SENT, SYN_RECEIVED, ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2, TIME_WAIT, CLOSING, CLOSE_WAIT, LAST_ACK, and 2MSL. Transitions occur based on the exchange of SYN, ACK, FIN, and RST flags.

Conclusion

The article covered network layering, packet encapsulation, TCP header fields, the three‑way handshake, four‑way termination, and common anomalies during connection setup and teardown. Understanding these concepts is crucial for diagnosing network issues and building robust client‑server applications.