A Linux Network Latency Diagnosis Method That Eliminates Overtime
This article walks through a step‑by‑step Linux workflow—using hping3, traceroute, wrk, tcpdump, and Wireshark—to pinpoint the root causes of network latency, illustrate delayed TCP ACK effects, and verify findings with concrete command outputs and packet captures.
In Linux servers, kernel tuning, DPDK, and XDP can improve attack resistance, while application‑level caches, WAF, and CDN mitigate DDoS impact; however, once DDoS traffic reaches the server, network latency typically spikes regardless of application optimizations.
Linux Network Latency
Network latency (RTT) measures the round‑trip time of packets between source and destination. Application latency adds the time a server spends processing a request, effectively RTT plus processing time.
Because many services disable ICMP ping, the article recommends using traceroute or hping3 in TCP/UDP mode to measure latency.
# -c: 3 requests
# -S: Set TCP SYN
# -p: Set port to 80
$ hping3 -c 3 -S -p 80 google.com
HPING google.com (eth0 142.250.64.110): S set, 40 headers + 0 data bytes
len=46 ip=142.250.64.110 ttl=51 id=47908 sport=80 flags=SA seq=0 win=8192 rtt=9.3 ms
len=46 ip=142.250.64.110 ttl=51 id=6788 sport=80 flags=SA seq=1 win=8192 rtt=10.9 ms
len=46 ip=142.250.64.110 ttl=51 id=37699 sport=80 flags=SA seq=2 win=8192 rtt=11.9 ms
--- baidu.com hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 9.3/10.9/11.9 msTraceroute example:
$ traceroute --tcp -p 80 -n google.com
traceroute to google.com (142.250.190.110), 30 hops max, 60 byte packets
1 * * *
2 240.1.236.34 0.198 ms * *
3 * * 243.254.11.5 0.189 ms
4 * 240.1.236.17 0.216 ms 240.1.236.24 0.175 ms
5 241.0.12.76 0.181 ms 108.166.244.15 0.234 ms 241.0.12.76 0.219 ms
...
24 142.250.190.110 17.465 ms 108.170.244.1 18.532 ms 142.251.60.207 18.595 msCase Study
Two hosts are prepared:
host1 (192.168.0.30): runs two Nginx containers—one official and one built with artificial latency.
host2 (192.168.0.2): analysis host.
host1 preparation
Run the containers:
# Official nginx
$ docker run --network=host --name=good -itd nginx
fb4ed7cb9177d10e270f8320a7fb64717eac3451114c9fab3c50e02be2e88ba2
# Latency version of nginx
$ docker run --name nginx --network=host -itd feisky/nginx:latency
b99bd136dcfd907747d9c803fdc0255e578bad6d66f4e9c32b826d75b6812724Verify both containers serve traffic:
$ curl http://127.0.0.1
<!DOCTYPE html>
<html>
<p><em>Thank you for using nginx.</em></p>
</html>
$ curl http://127.0.0.1:8080
<!DOCTYPE html>
<html>
<p><em>Thank you for using nginx.</em></p>
</html>host2 preparation
Test latency with hping3:
$ hping3 -c 3 -S -p 80 192.168.0.30
HPING 192.168.0.30 (eth0 192.168.0.30): S set, 40 headers + 0 data bytes
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=80 flags=SA seq=0 win=29200 rtt=7.8 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=80 flags=SA seq=1 win=29200 rtt=7.7 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=80 flags=SA seq=2 win=29200 rtt=7.6 ms
--- 192.168.0.30 hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 7.6/7.7/7.8 msPort 8080:
# Test 8080 latency
$ hping3 -c 3 -S -p 8080 192.168.0.30
HPING 192.168.0.30 (eth0 192.168.0.30): S set, 40 headers + 0 data bytes
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=8080 flags=SA seq=0 win=29200 rtt=7.7 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=8080 flags=SA seq=1 win=29200 rtt=7.6 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=8080 flags=SA seq=2 win=29200 rtt=7.3 ms
--- 192.168.0.30 hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 7.3/7.6/7.7 msSingle‑request latency is similar, but concurrent load reveals differences. Using wrk (https://github.com/wg/wrk):
$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30/
Running 10s test @ http://192.168.0.30/
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 9.19ms 12.32ms 319.61ms 97.80%
Req/Sec 6.20k 426.80 8.25k 85.50%
Latency Distribution
50% 7.78ms
75% 8.22ms
90% 9.14ms
99% 50.53ms
123558 requests in 10.01s, 100.15MB read
Requests/sec: 12340.91
Transfer/sec: 10.00MB
$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
Running 10s test @ http://192.168.0.30:8080/
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 43.60ms 6.41ms 56.58ms 97.06%
Req/Sec 1.15k 120.29 1.92k 88.50%
Latency Distribution
50% 44.02ms
75% 44.33ms
90% 47.62ms
99% 48.88ms
22853 requests in 10.01s, 18.55MB read
Requests/sec: 2283.31
Transfer/sec: 1.85MBThe official Nginx (port 80) averages ~9 ms latency, while the latency‑modified Nginx (port 8080) averages ~44 ms, with 50 % of requests already exceeding 44 ms.
To investigate, capture packets on host1: $ tcpdump -nn tcp port 8080 -w nginx.pcap After the wrk run, stop tcpdump, copy the nginx.pcap file to a machine with Wireshark, and open it.
In Wireshark, filter by a specific TCP stream (e.g., tcp.stream eq 24) and view the Flow Graph (Statistics → Flow Graph, TCP Flows). The graph shows that the first three handshakes and the first HTTP request are fast, but the second HTTP request experiences a 40 ms delay before the client sends an ACK.
This 40 ms pause corresponds to the TCP delayed ACK timeout. TCP does not acknowledge every packet immediately; it waits (typically up to 40 ms) to see if there is outbound data to piggyback the ACK onto. If no data arrives, the ACK is sent after the timeout.
The client in this test is the wrk tool, which by default uses delayed ACK. The article cites the Linux documentation for TCP_QUICKACK (since Linux 2.4.4) that enables immediate ACKs when set, otherwise delayed ACK is used.
TCP_QUICKACK (since Linux 2.4.4)
Enable quickack mode if set or disable quickack mode if cleared.
In quickack mode, acks are sent immediately, rather than delayed if needed.
This flag is not permanent; subsequent TCP operation may re‑enter delayed ACK based on internal processing.To confirm the hypothesis, run strace on wrk:
$ strace -f wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
... setsockopt(52, SOL_TCP, TCP_NODELAY, [1], 4) = 0 ...The output shows that wrk sets TCP_NODELAY but does not enable TCP_QUICKACK, confirming that delayed ACK is responsible for the extra latency.
Conclusion
The article demonstrates a systematic approach to diagnosing increased network latency on Linux:
Use hping3 and wrk to verify latency for single and concurrent requests.
Use traceroute to confirm routing paths and per‑hop delays.
Capture traffic with tcpdump and analyze it in Wireshark.
Inspect socket options with strace to detect delayed ACK behavior.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
Linux Tech Enthusiast
Focused on sharing practical Linux technology content, covering Linux fundamentals, applications, tools, as well as databases, operating systems, network security, and other technical knowledge.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.