Wireshark Packet Capture Mastery: Decoding TCP Handshakes, Retransmissions & Resets

Capture Setup and Quality Verification

The article begins by establishing a fictional teaching topology: client 192.0.2.10 accessing server 192.0.2.20:443, with the capture file named incident.pcapng. It emphasizes that capture quality must be verified before analysis: list interfaces with tshark -D, confirm routing with ip route get 192.0.2.20, and capture with time limits and endpoint filters using dumpcap or tcpdump. Key commands include:

sudo dumpcap -i eth0 -f 'host 192.0.2.20 and tcp port 443' -a duration:60 -w incident.pcapng

Ring-buffer capture for unpredictable faults:

sudo dumpcap -i eth0 -f 'tcp port 443' -b filesize:100000 -b files:5 -w ring.pcapng

. Snapshot length ( -s 0) should be full packet for TLS/HTTP reassembly. Capture filters (libpcap syntax) differ from display filters (Wireshark field syntax); the article shows both for the same endpoint condition.

Capture Metadata and Clock Synchronization

File metadata via capinfos incident.pcapng reveals capture duration, packet count, average rate — but average rate does not prove zero loss. Clock synchronization between endpoints is critical: timedatectl status and chronyc tracking check host time sync; even with sync, microsecond accuracy is not guaranteed, so sequence/ack numbers are preferred over absolute timestamps for correlating segments.

Finding Complete Connections and Handshake Analysis

TCP conversation statistics ( tshark -r incident.pcapng -q -z conv,tcp) identify long-lived or high-volume flows. Handshake steps are isolated with display filters:

Client SYN: tcp.flags.syn == 1 && tcp.flags.ack == 0 Server SYN-ACK: tcp.flags.syn == 1 && tcp.flags.ack == 1 Final ACK (within stream 0): tcp.stream == 0 && tcp.flags.ack == 1 && tcp.flags.syn == 0 Stream index ( tcp.stream == 0) groups bidirectional packets. Exporting key fields (relative time, addresses, ports, flags, sequence, ack, length) via TShark produces a timeline. A fictional handshake timeline illustrates relative sequence numbers:

0.000000 client -> server SYN      seq=0
0.012000 server -> client SYN,ACK  seq=0 ack=1
0.024000 client -> server ACK      seq=1 ack=1

Raw sequence numbers ( tcp.seq_raw, tcp.ack_raw) enable cross-capture correlation. SYN retransmissions without response require checking server capture, listen state, and return path — not assuming a single cause.

Retransmission Analysis: Marks vs. Evidence

The article distinguishes analyzer marks from root causes: tcp.analysis.retransmission — candidate retransmission; could be capture loss, mid-capture start, or duplicate mirroring. tcp.analysis.fast_retransmission — often with duplicate ACKs; behavior varies by congestion control and SACK. tcp.analysis.duplicate_ack — receiver re-ACKs same cumulative sequence; may indicate loss, reordering, window changes, or capture artifacts. tcp.analysis.out_of_order — arrival order mismatch; could be network reordering, multi-interface capture, or mirror duplicates. tcp.analysis.lost_segment — sequence gap seen by analyzer; may be capture drop, not network loss. tcp.analysis.spurious_retransmission — segment resent after analyzer saw ACK; could be ACK loss, timing issues, or mirror duplicates.

Retransmission analysis must keep bidirectional flow; stripping ACKs breaks diagnosis. Exporting candidate fields ( frame.number, frame.time_relative, ip.src, tcp.seq_raw, tcp.len, tcp.ack_raw) aids manual verification. SACK blocks ( tcp.options.sack_le || tcp.options.sack_re) show non-contiguous received ranges. Filtering a specific raw sequence range (

tcp.stream == 0 && tcp.seq_raw >= 100000 && tcp.seq_raw < 102000

) checks first transmission vs. retransmission coverage.

Connection Reset (RST) Diagnosis

RST packets ( tcp.flags.reset == 1) require direction, sequence validity, and connection state context. Source IP may appear as server but could be proxy, firewall, or load balancer. Separate filters for server-side ( ip.src == 192.0.2.20 && tcp.flags.reset == 1) and client-side ( ip.src == 192.0.2.10 && tcp.flags.reset == 1) resets. Export RST metadata ( frame.number, frame.time_relative, ip.src, tcp.srcport, ip.dst, tcp.dstport, tcp.seq_raw, tcp.ack_raw) for audit trails.

SYN followed by immediate RST suggests no listener or policy reject; verify with sudo ss -lntp 'sport = :443'. Mid-stream RST correlates with application logs (

journalctl -u app.service --since '2026-09-30 10:00:00' --until '2026-09-30 10:10:00' --no-pager

). FIN ( tcp.flags.fin == 1) indicates graceful close; half-close and ACK+FIN merging mean four packets are not guaranteed. Idle connection reuse after proxy/NAT timeout causes RST; compare connection pool, proxy, and NAT idle timers. TTL, IP ID, and flags (

tshark -r incident.pcapng -Y 'tcp.stream == 0' -T fields -e frame.number -e ip.src -e ip.ttl -e ip.id -e tcp.flags

) help differentiate reset sources but are not definitive.

Window, RTT, and Throughput Analysis

Zero window ( tcp.analysis.zero_window) signals receiver cannot accept data; causes include application read stalls, memory pressure. Zero window probes ( tcp.analysis.zero_window_probe) test window reopening. Window updates ( tcp.analysis.window_update) show receiver capacity changes; window scaling negotiated in handshake — missing handshake breaks scaling interpretation. Window full ( tcp.analysis.window_full) indicates sender near receiver's advertised limit.

Export window fields:

tshark -r incident.pcapng -Y 'tcp.stream == 0' -T fields -e frame.time_relative -e ip.src -e tcp.window_size_value -e tcp.window_size_scalefactor -e tcp.window_size

. ACK RTT ( tcp.analysis.ack_rtt) is analyzer's estimate; affected by capture position, delayed ACK, retransmission. Export:

tshark -r incident.pcapng -Y 'tcp.analysis.ack_rtt' -T fields -e frame.time_relative -e tcp.stream -e tcp.analysis.ack_rtt

.

Bandwidth-delay product (BDP) example: 1 Gbit/s × 40 ms RTT = 5 MB in-flight data needed. Python snippet calculates BDP. IO statistics per second ( tshark -r incident.pcapng -q -z io,stat,1) show traffic volume but include retransmissions and headers. Endpoint kernel state via ss -ti dst 192.0.2.20 supplements capture with TCP stats (retransmits, buffer sizes).

Eliminating Capture Artifacts

Offload features (TSO, LRO) make host captures show segments larger than wire MTU; check with sudo ethtool -k eth0. Checksum offload may show bad checksums in host capture ( tcp.checksum.status == 0); verify at receiver or external tap. Truncation ( frame.cap_len < frame.len) means payload missing — cannot infer sender didn't transmit. Capture drops from ip -s link show dev eth0 and pidstat -u -C dumpcap 1 5 separate NIC drops, kernel buffer drops, and network drops. Duplicate packets from any interface or mirrored ports: export frame.interface_id with sequence/time to identify duplicates. Merging captures ( mergecap -w combined.pcapng client.pcapng server.pcapng) does not fix clock skew, NAT, or duplicates; prefer raw sequence correlation. Time-slicing (

editcap -A '2026-09-30 10:00:00' -B '2026-09-30 10:05:00' incident.pcapng window.pcapng

) must include handshake context. Saving a full stream (

tshark -r incident.pcapng -Y 'tcp.stream == 0' -w selected-stream.pcapng

) preserves analyzer context. IP fragmentation ( ip.flags.mf == 1 || ip.frag_offset > 0) can affect reassembly.

Post-Handshake: TLS and Application Layer

DNS ( dns) must resolve before TCP. TLS handshake ( tls.handshake.type == 1 || tls.handshake.type == 2 for ClientHello/ServerHello) occurs over TCP; TCP success ≠ TLS success. TLS alerts ( tls.alert_message) may explain early closure. Plain HTTP ( http.request || http.response) only visible in cleartext or at TLS termination point. Export HTTP response codes and timing:

tshark -2 -r incident.pcapng -Y 'http.response' -T fields -e frame.number -e tcp.stream -e http.response.code -e http.time

. Follow TCP stream ( tshark -r incident.pcapng -q -z follow,tcp,hex,0) shows raw bytes; encrypted streams appear opaque. Pure ACKs (

tcp.stream == 0 && tcp.len == 0 && tcp.flags.ack == 1 && tcp.flags.syn == 0 && tcp.flags.fin == 0 && tcp.flags.reset == 0

) confirm TCP receipt, not application processing. HTTP/2 ( http2) multiplexes requests on one TCP connection; TCP stream ≠ request stream. HTTP/3 uses QUIC/UDP ( quic || udp.port == 443); absence of TCP flows does not mean no connection.

Verifiable Statistics, Not Red Alerts

Protocol hierarchy ( tshark -r incident.pcapng -q -z io,phs) shows protocol distribution. Expert info ( tshark -r incident.pcapng -q -z expert) lists analyzer flags; severity ≠ business impact. Retransmission candidate counts must define merge rules, duplicate exclusion, and denominator. Example:

tshark -r incident.pcapng -Y 'tcp.stream == 0 && (tcp.analysis.retransmission || tcp.analysis.fast_retransmission || tcp.analysis.spurious_retransmission)' -T fields -e frame.number | wc -l

. Unique streams with RST:

tshark -r incident.pcapng -Y 'tcp.flags.reset == 1' -T fields -e tcp.stream | sort -n -u | wc -l

. Directional payload bytes (including retransmissions):

tshark -r incident.pcapng -Y 'tcp.stream == 0 && ip.src == 192.0.2.10' -T fields -e tcp.len | awk 'NF {n+=$1} END {print n+0}'

. RTT percentiles via exported ACK RTT values sorted and computed with nearest-rank method (Python script). Field availability checked via

tshark -G fields | awk -F'\t' '$1=="F" && $3 ~ /^tcp\.(analysis|window|seq|ack)/ {print $3,$2}'

. Preferences (relative sequence numbers, sequence analysis, TCP reassembly) exported for reproducibility:

tshark -G currentprefs | grep -E 'tcp\.(relative_sequence_numbers|analyze_sequence_numbers|desegment_tcp_streams)'

.

Path MTU, Congestion Signals, Endpoint Resources

ICMPv4 Fragmentation Needed ( icmp.type == 3 && icmp.code == 4) indicates PMTU black hole. ICMPv6 Packet Too Big ( icmpv6.type == 2) serves same role; blocking all ICMPv6 breaks PMTUD. MSS from SYN options (

tshark -r incident.pcapng -Y 'tcp.flags.syn == 1' -T fields -e frame.number -e ip.src -e tcp.options.mss_val

) shows endpoint segment size; tunnels may reduce effective MTU. ECN flags ( tcp.flags.ece == 1 || tcp.flags.cwr == 1) and IP ECN CE ( ip.dsfield.ecn == 3) signal congestion explicitly. Connection summary ( ss -s) and kernel TCP stats ( nstat -az) provide cumulative counters; delta over incident window needed. File integrity via SHA-256 ( sha256sum incident.pcapng selected-stream.pcapng). Evidence checklist (YAML) records capture point, handshake presence, drop counts, bidirectional visibility, offload review, clock error bound, and confirmed observations.

Three Fictional Fault Walkthroughs

SYN no SYN-ACK : Client retransmits SYN. Three branches: (a) server capture sees nothing → check routing, firewall, NAT, LB ingress; (b) server sees SYN but no SYN-ACK → check listen socket, host policy, resources; (c) server sends SYN-ACK but client doesn't see → check return path, client-side filter. Each branch has different ownership.

Slow download with duplicate ACKs, SACK gaps, fast retransmit : First verify capture drops; then compare receiver capture for same sequence gaps. If sender's segment missing at receiver and later retransmission fills gap, path loss is likely. Correlate with interface counters, queue stats, congestion signals, device logs.

Idle long-lived connection reset on reuse : Compare connection pool reuse time, proxy idle timeout, NAT session timeout. If proxy expires at 30s but client pool keeps 5min, reuse fails. Fix: client pre-eviction, keepalives, or unified timeout — not blindly extending all timeouts.

Additional scenario: Data ACKed quickly but application response delayed seconds later. TCP ACK only proves kernel receipt; investigate application queue, thread pool, DB, external dependencies. Zero window suggests receiver read stall; normal window points to application-layer delay. Packet analysis bounds the wait stage but cannot pinpoint code.

Reporting Standards

A sound capture conclusion documents: capture location, time range, filters, handshake inclusion, capture drops, visible protocol layers. Evidence statements cite specific observations (e.g., "client retransmitted same sequence range at these timestamps", "server sent RST 3 seconds after receiving request", "window persisted at zero from server direction"). Separate observations from inferences; list verified root causes or next required evidence.

References

Wireshark TCP Analysis Documentation

TShark Manual

dumpcap Manual

TCP Display Filter Fields

Wireshark TCP Stream Graph