How to Diagnose and Reduce Linux Network Latency with hping3, wrk, and Wireshark

Linux servers can suffer increased network latency due to DDoS attacks, kernel processing speed, or application‑level bottlenecks. While kernel tuning, DPDK, or XDP can harden the server, the article focuses on practical methods to measure and diagnose latency at both the single‑request and concurrent‑request levels.

Network Latency Basics

Network latency (RTT) is the time for a packet to travel from source to destination and back. Application latency adds the time the server spends processing the request. The ping command (ICMP‑based) is commonly used to obtain RTT, but many services disable ICMP, requiring traceroute or hping3 in TCP/UDP mode.

Measuring RTT with hping3

# -c: 3 requests
# -S: Set TCP SYN
# -p: Set port to 80
$ hping3 -c 3 -S -p 80 google.com
HPING google.com (eth0 142.250.64.110): S set, 40 headers + 0 data bytes
len=46 ip=142.250.64.110 ttl=51 id=47908 sport=80 flags=SA seq=0 win=8192 rtt=9.3 ms
len=46 ip=142.250.64.110 ttl=51 id=6788 sport=80 flags=SA seq=1 win=8192 rtt=10.9 ms
len=46 ip=142.250.64.110 ttl=51 id=37699 sport=80 flags=SA seq=2 win=8192 rtt=11.9 ms
--- baidu.com hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 9.3/10.9/11.9 ms

When ICMP is blocked, traceroute can be used in TCP mode:

$ traceroute --tcp -p 80 -n google.com
traceroute to google.com (142.250.190.110), 30 hops max, 60 byte packets
 1  * * *
 2  240.1.236.34  0.198 ms * *
 3  * 243.254.11.5  0.189 ms
 ...
24  142.250.190.110  17.465 ms 108.170.244.1  18.532 ms 142.251.60.207  18.595 ms

Case Study: Docker‑Based Nginx Latency Test

Two hosts are prepared:

host1 (192.168.0.30) runs two containers: the official Nginx on port 80 and a latency‑modified Nginx on port 8080.

host2 (192.168.0.2) acts as the analysis machine.

Launching the containers on host1

# Official nginx
$ docker run --network=host --name=good -itd nginx
# Latency version of nginx
$ docker run --name nginx --network=host -itd feisky/nginx:latency

Verification with curl shows both containers respond.

Measuring single‑request latency with hping3

# Port 80
$ hping3 -c 3 -S -p 80 192.168.0.30
# Port 8080 (latency Nginx)
$ hping3 -c 3 -S -p 8080 192.168.0.30

Both ports show roughly 7 ms RTT for a single request.

Measuring concurrent latency with wrk

# 80 (official Nginx)
$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30/
Latency   Avg: 9.19 ms  Stdev: 12.32 ms  Max: 319.61 ms
Requests/sec: 12340.91

# 8080 (latency Nginx)
$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
Latency   Avg: 43.60 ms  Stdev: 6.41 ms  Max: 56.58 ms
Requests/sec: 2283.31

The latency‑modified Nginx exhibits a four‑fold increase in average latency under load.

Deep Packet Inspection with tcpdump and Wireshark

On host1, capture traffic for the 8080 service: $ tcpdump -nn tcp port 8080 -w nginx.pcap After the wrk run, stop tcpdump and open nginx.pcap in Wireshark. Filtering by the TCP stream (e.g., tcp.stream eq 24) reveals the handshake and HTTP exchanges. The flow graph shows that the second HTTP request experiences a 40 ms delay before the client sends an ACK.

This 40 ms pause matches the default TCP delayed ACK timeout, indicating that the client (the wrk tool) is using the delayed acknowledgment mechanism.

TCP Delayed ACK vs. Quick ACK

Linux enables delayed ACK by default. Only when a socket sets TCP_QUICKACK does the kernel send ACKs immediately. The article shows that wrk only sets TCP_NODELAY (disabling Nagle’s algorithm) but does not enable TCP_QUICKACK, so the extra 40 ms delay is expected.

$ strace -f wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
... setsockopt(52, SOL_TCP, TCP_NODELAY, [1], 4) = 0 ...

Conclusion and Recommendations

Network latency can stem from transport‑layer delays (e.g., TCP delayed ACK), kernel packet‑processing speed, or application design. To diagnose and mitigate latency:

Use hping3 and wrk to measure single‑request and concurrent latency.

Employ traceroute to verify routing paths and per‑hop delays.

Capture traffic with tcpdump and analyze with Wireshark to spot protocol‑level issues such as delayed ACK.

Inspect socket options with strace to ensure appropriate flags ( TCP_QUICKACK, TCP_NODELAY) are set for latency‑sensitive workloads.