Diagnosing Linux Network Latency with hping3, traceroute, and Wireshark
This article explains how to identify the root causes of increased network latency on Linux servers by using tools such as hping3, traceroute, wrk, tcpdump, and Wireshark, demonstrating step‑by‑step measurements, analysis of TCP delayed acknowledgments, and practical recommendations for mitigating latency in production environments.
In Linux servers, kernel tuning, DPDK, and XDP can improve resistance to attacks and reduce DDoS impact on normal services. Applications can use caches, WAF, CDN, etc., to mitigate DDoS effects.
However, if DDoS traffic reaches the Linux server, application‑level optimizations cannot prevent significantly increased network latency.
Therefore, in practice we often combine Linux servers with professional traffic‑scrubbing and firewall devices.
Besides DDoS‑induced latency, many other factors can cause network delay, such as slow network transmission, slow Linux kernel protocol stack processing, and slow application data handling.
When encountering these causes, how can we locate the root cause of network latency? Let’s discuss network latency in this article.
Linux Network Latency
Network latency (RTT) refers to the round‑trip time for data to travel from source to destination and back.
Application latency is the time an application takes to receive a request and return a response, essentially the sum of network transmission time and data processing time.
People usually use the ping command to test network latency; ping is based on ICMP and measures the time difference between request and response packets. Because it requires no authentication, many network attacks exploit it (e.g., nmap, hping3).
To avoid these issues, many services disable ICMP, making ping unusable. In such cases, traceroute or hping3 in TCP/UDP mode can be used to obtain latency.
Example:
# -c: 3 requests
# -S: Set TCP SYN
# -p: Set port to 80
$ hping3 -c 3 -S -p 80 google.com
HPING google.com (eth0 142.250.64.110): S set, 40 headers + 0 data bytes
len=46 ip=142.250.64.110 ttl=51 id=47908 sport=80 flags=SA seq=0 win=8192 rtt=9.3 ms
len=46 ip=142.250.64.110 ttl=51 id=6788 sport=80 flags=SA seq=1 win=8192 rtt=10.9 ms
len=46 ip=142.250.64.110 ttl=51 id=37699 sport=80 flags=SA seq=2 win=8192 rtt=11.9 ms
--- baidu.com hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 9.3/10.9/11.9 msOr using traceroute:
$ traceroute --tcp -p 80 -n google.com
traceroute to google.com (142.250.190.110), 30 hops max, 60 byte packets
1 * * *
2 240.1.236.34 0.198 ms * *
3 * * 243.254.11.5 0.189 ms
4 * 240.1.236.17 0.216 ms 240.1.236.24 0.175 ms
5 241.0.12.76 0.181 ms 108.166.244.15 0.234 ms 241.0.12.76 0.219 ms
...
24 142.250.190.110 17.465 ms 108.170.244.1 18.532 ms 142.251.60.207 18.595 mstraceroutesends three packets per hop and reports RTT; a timeout results in a ‘*’.
Case Study
We will host two machines: host1 (192.168.0.30) running two Nginx web applications (normal and latency‑induced) and host2 (192.168.0.2) for analysis.
host1 (192.168.0.30): official Nginx on port 80 and a latency‑modified Nginx on port 8080.
host2 (192.168.0.2): analysis host.
host1 preparation
Run two containers on host1:
# Official nginx
$ docker run --network=host --name=good -itd nginx
# Latency version of nginx
$ docker run --name nginx --network=host -itd feisky/nginx:latencyVerify both containers serve traffic:
$ curl http://127.0.0.1
<!DOCTYPE html>
<html>
...<p><em>Thank you for using nginx.</em></p>
</html>
$ curl http://127.0.0.1:8080
...<p><em>Thank you for using nginx.</em></p>host2 preparation
Test latency on ports 80 and 8080 using hping3:
Port 80:
$ hping3 -c 3 -S -p 80 192.168.0.30
HPING 192.168.0.30 (eth0 192.168.0.30): S set, 40 headers + 0 data bytes
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=80 flags=SA seq=0 win=29200 rtt=7.8 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=80 flags=SA seq=1 win=29200 rtt=7.7 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=80 flags=SA seq=2 win=29200 rtt=7.6 ms
--- 192.168.0.30 hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 7.6/7.7/7.8 msPort 8080:
# Test latency on port 8080
$ hping3 -c 3 -S -p 8080 192.168.0.30
HPING 192.168.0.30 (eth0 192.168.0.30): S set, 40 headers + 0 data bytes
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=8080 flags=SA seq=0 win=29200 rtt=7.7 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=8080 flags=SA seq=1 win=29200 rtt=7.6 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=8080 flags=SA seq=2 win=29200 rtt=7.3 ms
--- 192.168.0.30 hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 7.3/7.6/7.7 msBoth ports show similar latency (~7 ms) for single requests. Next, we test concurrent requests with wrk:
Port 80:
$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30/
Running 10s test @ http://192.168.0.30/
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 9.19ms 12.32ms 319.61ms 97.80%
Req/Sec 6.20k 426.80 8.25k 85.50%
Latency Distribution
50% 7.78ms
75% 8.22ms
90% 9.14ms
99% 50.53ms
123558 requests in 10.01s, 100.15MB read
Requests/sec: 12340.91
Transfer/sec: 10.00MBPort 8080:
$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
Running 10s test @ http://192.168.0.30:8080/
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 43.60ms 6.41ms 56.58ms 97.06%
Req/Sec 1.15k 120.29 1.92k 88.50%
Latency Distribution
50% 44.02ms
75% 44.33ms
90% 47.62ms
99% 48.88ms
22853 requests in 10.01s, 18.55MB read
Requests/sec: 2283.31
Transfer/sec: 1.85MBThe official Nginx (port 80) averages ~9 ms latency, while the latency‑modified Nginx (port 8080) averages ~44 ms, with 50 % of requests already at 44 ms.
To investigate, we capture packets on host1 with tcpdump: $ tcpdump -nn tcp port 8080 -w nginx.pcap After running wrk on host2, we stop tcpdump and open the capture in Wireshark. Filtering by the TCP stream reveals the handshake and request/response timings. The first HTTP request is fast, but the second experiences a ~40 ms delay before the client sends an ACK.
This 40 ms corresponds to TCP delayed ACK timeout, an optimization where the TCP stack waits (≈40 ms) to see if there is data to piggyback the ACK onto.
We verify that wrk sets TCP_NODELAY but does not enable TCP_QUICKACK, confirming the delayed ACK behavior.
$ strace -f wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
... setsockopt(52, SOL_TCP, TCP_NODELAY, [1], 4) = 0 ...Conclusion
This article demonstrates how to analyze increased network latency. Network latency is a core performance metric affected by transmission, packet processing, and other factors. Excessive latency degrades user experience.
Use hping3 and wrk to verify latency for single and concurrent requests.
Use traceroute to confirm routing and per‑hop latency.
Use tcpdump and Wireshark to verify packet flow.
Use strace to observe socket calls made by applications.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
Open Source Linux
Focused on sharing Linux/Unix content, covering fundamentals, system development, network programming, automation/operations, cloud computing, and related professional knowledge.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.