Debugging a TCP Packet Length Bug That Blocks Production at 70% Progress

Background

A device registration process communicates with the server over TCP, sending a configuration file (key=value) in the D4 stage and a secret key in the D5 stage. When the name field in the config is rabbit , production completes; when it is rabbit‑TD , the progress stops at 70%.

Investigation Process

2.1 Check code – No length validation on the name field was found on either side.

2.2 Server logs – Only D4 logs appear; D5 logs are missing, suggesting the device does not send D5 data.

2.3 Server packet capture – Using Microsoft Network Monitor and Wireshark, the server receives the D4 packet and sends an ACK, but no separate D5 packet is observed.

2.4 Device packet capture – A tcpdump -i fetho host 192.168.1.253 capture shows that D4 and D5 are actually combined into a single TCP segment, which explains why the server never sees a distinct D5 packet.

2.5 Further server capture – The combined packet contains both the configuration file and the secret key, confirming the merging.

Packet Format and Analysis

Each stage follows the format: 0x1234abcd, length, type, data . For name=rabbit the length fields are correct (D4 length = 0x00 0x00 0x03 0xF4 = 1011 bytes, D5 length = 0x00 0x00 0x01 0x00 = 256 bytes) and the total fits within the 1024‑byte read buffer.

For name=rabbit‑TD the D4 length becomes 0x00 0x00 0x03 0xF6 = 1014 bytes. When the server reads the first 1024 bytes, it consumes the first byte of the D5 length field, causing the remaining three bytes of the length and the type byte to be mis‑aligned. The server then interprets the length as 65538 (0x00 0x01 0x00 0x02), which does not match the actual 256‑byte payload, leading to a parsing error and aborting the D5 processing.

Root Cause

The bug originates from the server’s fixed 1024‑byte read window: when the combined D4/D5 packet reaches exactly 1024 bytes, the length field of D5 is split across two reads, and the first byte is lost, resulting in an incorrect length calculation.

Solutions

Solution 1 – Preserve the first byte of the D5 length when performing the second read, ensuring the full 4‑byte length is reconstructed before parsing.

Solution 2 – Pad the D4 configuration file so that the combined packet size pushes the D5 length field beyond the 1024‑byte boundary (e.g., make the D4 payload 1015 or 1019 bytes). This forces the entire length field to be read in a single operation, which was verified to work with names like Rabbit‑TDDDDDDD and Rabbit‑TDD .