Understanding TCP RST Attacks: From Handshakes to Exploiting Sequence Numbers

What is TCP?

TCP is a transport‑layer protocol that sits above IP and provides reliable, connection‑oriented byte‑stream communication between ports (processes). It ensures reliability through acknowledgments, retransmissions, and flow control, and it reassembles out‑of‑order IP packets back into the original byte stream.

Three‑Way Handshake

The connection is established by a SYN, SYN‑ACK, ACK exchange. The SYN flag is set to 1, and the initial sequence number (ISN) is chosen based on the operating system’s algorithm, often related to system time. The client also advertises its maximum segment size (MSS) and initial window size.

Sliding Window

The sliding window mechanism allows the sender to transmit multiple packets before receiving individual acknowledgments, as long as the difference between the next sequence number and the smallest unacknowledged sequence number stays within the advertised window size. This improves throughput by reducing waiting time.

Four‑Way Connection Termination

Normal connection closure uses a four‑step FIN/ACK exchange. The FIN flag signals a graceful shutdown; the side that sends FIN must wait until all buffered data is transmitted before sending the FIN packet, unlike the abrupt RST termination.

RST Flag and Its Role

The RST (reset) flag indicates an abnormal termination. When a host sends an RST packet, it discards any buffered data and does not require an ACK. Typical scenarios include receiving a packet for a closed port or encountering an unexpected state.

RST Attack Mechanics

An attacker (C) can forge a TCP packet that appears to come from an established client (A) and send it to the server (B). If the forged packet carries the RST flag, B will immediately drop the connection and any buffered data, effectively cutting off A’s session.

Packet Forgery Requirements

To succeed, the attacker must correctly guess the four‑tuple (source IP, source port, destination IP, destination port) and a sequence number that falls within the server’s current receive window. The source port may be inferred if the target OS uses predictable port allocation; otherwise, it can be brute‑forced.

The sequence number space is 32 bits (0‑4,294,967,295). If the receive window is, for example, 65,535 bytes, an attacker needs to send at most 65,537 packets to guarantee a sequence number that lands inside the window (4,294,967,296 ÷ 65,535 ≈ 65,537). Because an RST packet is only 40 bytes (IP header + TCP header), this brute‑force approach can be completed in a few seconds on a modest network.

When the source port is also predictable, the attack becomes trivial; otherwise, the attacker must combine port‑guessing with sequence‑number brute forcing, which still remains feasible against many operating systems.