Diagnosing Linux Network Latency with hping3, traceroute, and wrk

Linux servers can suffer from high network latency due to many factors, including DDoS attacks, slow kernel packet processing, or inefficient application handling. While kernel tuning, DPDK, and XDP can improve resilience, once attack traffic reaches the server the latency often spikes, so professional traffic‑scrubbing and firewall devices are typically required.

Understanding Network and Application Latency

Network latency (RTT) measures the round‑trip time for a packet to travel from source to destination and back. Application latency adds the time the application spends processing the request, effectively the sum of network RTT and processing time.

The ping command (ICMP‑based) is commonly used to measure RTT, but many services disable ICMP, making traceroute or hping3 (TCP/UDP mode) necessary alternatives.

Example: Measuring RTT with hping3

# -c: 3 requests
# -S: set TCP SYN flag
# -p: target port 80
$ hping3 -c 3 -S -p 80 google.com
HPING google.com (eth0 142.250.64.110): S set, 40 headers + 0 data bytes
len=46 ip=142.250.64.110 ttl=51 id=47908 sport=80 flags=SA seq=0 win=8192 rtt=9.3 ms
len=46 ip=142.250.64.110 ttl=51 id=6788  sport=80 flags=SA seq=1 win=8192 rtt=10.9 ms
len=46 ip=142.250.64.110 ttl=51 id=37699 sport=80 flags=SA seq=2 win=8192 rtt=11.9 ms
--- baidu.com hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 9.3/10.9/11.9 ms

Using traceroute in TCP mode provides per‑hop RTT information:

$ traceroute --tcp -p 80 -n google.com
traceroute to google.com (142.250.190.110), 30 hops max, 60 byte packets
 1  * * *
 2  240.1.236.34  0.198 ms * *
 3  * * 243.254.11.5  0.189 ms
 ...
24  142.250.190.110  17.465 ms 108.170.244.1  18.532 ms 142.251.60.207  18.595 ms

Case Study: Host1 (two Nginx containers) and Host2 (analysis)

Host1 runs two Docker containers: the official Nginx on port 80 and a custom “latency” Nginx on port 8080.

# Official nginx
$ docker run --network=host --name=good -itd nginx
# Latency version of nginx
$ docker run --name nginx --network=host -itd feisky/nginx:latency

Both containers respond to curl requests, confirming they are serving traffic.

Measuring Single‑Request Latency with hping3

$ hping3 -c 3 -S -p 80 192.168.0.30
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=80 flags=SA rtt=7.8 ms
... (average ~7.7 ms)

# Test port 8080
$ hping3 -c 3 -S -p 8080 192.168.0.30
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=8080 flags=SA rtt=7.7 ms
... (average ~7.6 ms)

Single‑request latency is similar for both ports, but concurrent load reveals differences.

Concurrent Load Test with wrk

$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30/
Latency   9.19ms (avg) 12.32ms (stdev) 319.61ms (max)
Requests/sec: 12340.91

$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
Latency   43.60ms (avg) 6.41ms (stdev) 56.58ms (max)
Requests/sec: 2283.31

The latency version of Nginx shows roughly four times higher average latency under load.

Deep Packet Analysis with tcpdump and Wireshark

Capture traffic on host1: $ tcpdump -nn tcp port 8080 -w nginx.pcap After running wrk from host2, open the resulting nginx.pcap in Wireshark. Filter by a TCP stream (e.g., tcp.stream eq 24) and view the Flow Graph to see timing of each packet.

The Flow Graph shows that the second HTTP request experiences a ~40 ms delay before the client sends an ACK. This matches the TCP delayed‑ACK timeout.

Why Delayed ACK Happens

Linux defaults to delayed ACK, sending an acknowledgment only after a short timeout (≈40 ms) or when enough data is pending. The wrk client does not enable TCP_QUICKACK, so the delayed ACK mechanism is active.

TCP_QUICKACK (since Linux 2.4.4)
    Enable quickack mode if set; otherwise delayed ACK is used.
    ... (description omitted for brevity)

Confirming with strace shows that wrk sets only TCP_NODELAY and not TCP_QUICKACK:

$ strace -f wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
... setsockopt(52, SOL_TCP, TCP_NODELAY, [1], 4) = 0 ...

Conclusion

Use hping3 and wrk to verify latency for single requests and under concurrency.

Use traceroute to check routing and per‑hop delays.

Capture packets with tcpdump and analyze them in Wireshark to see protocol‑level timing.

Inspect socket options with strace to understand ACK behavior.