When FIN Packets Arrive Out of Order: Uncovering the Linux Kernel’s TCP Handshake Bug

Problem Overview

The classic TCP four‑way handshake can be disrupted by unusual packet ordering. In a real‑world case the server experienced a timeout because a FIN packet arrived after an ACK packet, causing the connection to close only after the client retransmitted the FIN.

Key Observations

Both client and server send FIN almost simultaneously.

Client receives the server's FIN first and replies with an ACK.

Server then receives the client’s ACK before its own FIN, leading to a mis‑ordered sequence.

The server fails to process the FIN, so the client waits for a timeout and retransmits.

Packet Trace Details

Important packet IDs: 14913, 14914, 20622, 20623 . The trace (shown below) highlights the sequence numbers and ACK numbers of each packet.

Upper half of the image represents the client, lower half the server.

20622 (seq=4416, ack=753) – client FIN.

14913 (seq=753, ack=4416) – server FIN.

20623 (seq=4417, ack=754) – client ACK for server FIN.

14914 (seq=754, ack=4416) – server ACK for client FIN.

Root Cause Analysis

The server processes ACK‑20623 correctly, moving from FIN_WAIT1 to FIN_WAIT2. When FIN‑20622 arrives, its ACK field (753) is smaller than the updated snd_una (754), so the kernel treats it as an old ACK and discards the FIN after sending a challenge ACK.

acceptable = tcp_ack(sk, skb, FLAG_SLOWPATH | FLAG_UPDATE_TS_RECENT | FLAG_NO_CHALLENGE_ACK) > 0;

If acceptable is false, the kernel executes:

tcp_send_challenge_ack(sk, skb);
/* discard the packet */

This matches the observed behavior: the FIN never reaches tcp_rcv_state_process() and is dropped.

eBPF Monitoring

Using ss and eBPF, the state transition was observed:

<idle>-0 [000] d.s. 42261.233808: changing from ESTABLISHED to FIN_WAIT1
<idle>-0 [000] dNs. 42261.464578: changing from FIN_WAIT1 to FIN_WAIT2

However, the transition to TIMEWAIT was not captured by eBPF because the kernel bypasses tcp_set_state() when moving directly to TIMEWAIT.

Kernel Code Investigation

The problematic path lies in tcp_rcv_state_process() where the ACK check rejects the FIN:

if (!acceptable) {
    tcp_send_challenge_ack(sk, skb);
    goto discard;
}

Further inspection of tcp_ack() shows the old‑ACK detection logic that flags the FIN as already acknowledged.

Reproduction Attempts

Initial attempts using close() on the client side did not reproduce the issue because close() sets the SOCK_DEAD flag, leading the kernel to skip the problematic path. Replacing close() with shutdown() (which does not set SOCK_DEAD) caused the bug to appear consistently.

Conclusion

The disorder is a legitimate kernel behavior: the ACK check discards a FIN whose ACK number is lower than the current snd_una. The discrepancy between production and test environments stems from the use of close() versus shutdown() for terminating the socket.