How to Diagnose and Fix Linux Network Latency Issues
This article explains how to identify the root causes of increased network latency on Linux servers, covering tools such as ping, traceroute, hping3, and wrk, demonstrating packet analysis with tcpdump and Wireshark, and showing how TCP delayed ACK and socket options affect response times.
In the previous article I showed how to simulate a DDoS attack and mitigate it. DDoS uses a massive number of forged requests that consume server resources, preventing normal user requests from being served.
On Linux servers you can improve resistance to attacks through kernel tuning, DPDK, XDP, and by using caches, WAF, CDN at the application level. However, once DDoS traffic reaches the server, network latency usually spikes dramatically.
Therefore, in practice we combine Linux servers with professional traffic‑scrubbing and firewall devices.
Besides DDoS‑induced latency, other common causes include slow network transmission, slow kernel protocol‑stack packet processing, and slow application data handling.
Linux Network Latency
Network latency (RTT) is the round‑trip time for a packet to travel from source to destination and back. Application latency adds the time the application spends processing the request.
Most people use the
pingcommand to measure RTT. Ping is ICMP‑based and simply measures the time between ICMP request and response packets. Because it requires no authentication, many network tools (e.g.,
nmap,
hping3) can exploit it.
When ICMP is disabled, you can use
tracerouteor the TCP/UDP modes of
hping3to obtain latency information.
<code># -c: 3 requests
# -S: Set TCP SYN
# -p: Set port to 80
$ hping3 -c 3 -S -p 80 google.com
HPING google.com (eth0 142.250.64.110): S set, 40 headers + 0 data bytes
len=46 ip=142.250.64.110 ttl=51 id=47908 sport=80 flags=SA seq=0 win=8192 rtt=9.3 ms
len=46 ip=142.250.64.110 ttl=51 id=6788 sport=80 flags=SA seq=1 win=8192 rtt=10.9 ms
len=46 ip=142.250.64.110 ttl=51 id=37699 sport=80 flags=SA seq=2 win=8192 rtt=11.9 ms
--- baidu.com hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 9.3/10.9/11.9 ms</code>Or using TCP mode of
traceroute:
<code>$ traceroute --tcp -p 80 -n google.com
traceroute to google.com (142.250.190.110), 30 hops max, 60 byte packets
1 * * *
2 240.1.236.34 0.198 ms * *
3 * * 243.254.11.5 0.189 ms
4 * 240.1.236.17 0.216 ms 240.1.236.24 0.175 ms
5 241.0.12.76 0.181 ms 108.166.244.15 0.234 ms 241.0.12.76 0.219 ms
...
24 142.250.190.110 17.465 ms 108.170.244.1 18.532 ms 142.251.60.207 18.595 ms</code>Case Study
We set up two hosts:
host1 (192.168.0.30): runs two Nginx containers – a normal one and a latency‑modified version.
host2 (192.168.0.2): analysis host.
host1 preparation
Run the containers:
<code># Official nginx
$ docker run --network=host --name=good -itd nginx
# Latency version of nginx
$ docker run --name nginx --network=host -itd feisky/nginx:latency</code>Verify they serve traffic:
<code>$ curl http://127.0.0.1
<!DOCTYPE html>
<html>
...<p><em>Thank you <strong>for</strong> using nginx.</em></p>
</html>
$ curl http://127.0.0.1:8080
...<p><em>Thank you <strong>for</strong> using nginx.</em></p>
</html></code>host2 preparation
Test latency with
hping3:
Port 80:
<code>$ hping3 -c 3 -S -p 80 192.168.0.30
HPING 192.168.0.30 (eth0 192.168.0.30): S set, 40 headers + 0 data bytes
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=80 flags=SA seq=0 win=29200 rtt=7.8 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=80 flags=SA seq=1 win=29200 rtt=7.7 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=80 flags=SA seq=2 win=29200 rtt=7.6 ms
--- 192.168.0.30 hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 7.6/7.7/7.8 ms</code>Port 8080:
<code># Test 8080 latency
$ hping3 -c 3 -S -p 8080 192.168.0.30
HPING 192.168.0.30 (eth0 192.168.0.30): S set, 40 headers + 0 data bytes
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=8080 flags=SA seq=0 win=29200 rtt=7.7 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=8080 flags=SA seq=1 win=29200 rtt=7.6 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=8080 flags=SA seq=2 win=29200 rtt=7.3 ms
--- 192.168.0.30 hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 7.3/7.6/7.7 ms</code>Single‑request latency is similar, but we need to see the effect under load. Using
wrk:
Port 80:
<code>$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30/
Running 10s test @ http://192.168.0.30/
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 9.19ms 12.32ms 319.61ms 97.80%
Req/Sec 6.20k 426.80 8.25k 85.50%
Latency Distribution
50% 7.78ms
75% 8.22ms
90% 9.14ms
99% 50.53ms
123558 requests in 10.01s, 100.15MB read
Requests/sec: 12340.91
Transfer/sec: 10.00MB</code>Port 8080:
<code>$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
Running 10s test @ http://192.168.0.30:8080/
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 43.60ms 6.41ms 56.58ms 97.06%
Req/Sec 1.15k 120.29 1.92k 88.50%
Latency Distribution
50% 44.02ms
75% 44.33ms
90% 47.62ms
99% 48.88ms
22853 requests in 10.01s, 18.55MB read
Requests/sec: 2283.31
Transfer/sec: 1.85MB</code>The official Nginx (port 80) averages ~9 ms latency, while the latency‑modified Nginx (port 8080) averages ~44 ms, with 90 % of requests taking far longer.
To investigate, capture packets on host1:
<code>$ tcpdump -nn tcp port 8080 -w nginx.pcap</code>After the
wrkrun, stop
tcpdumpand open the pcap in Wireshark. The following screenshots illustrate the TCP stream and flow graph.
The flow graph shows that the first HTTP request is fast, but the second experiences a ~40 ms delay after the server sends the first data packet. This corresponds to the TCP delayed‑ACK timeout.
TCP delayed ACK waits up to 40 ms to see if there is outbound data to piggyback the ACK. If the client (the
wrktool) does not enable
TCP_QUICKACK, the delayed ACK mechanism is used.
<code>TCP_QUICKACK (since Linux 2.4.4)
Enable quickack mode if set or disable if cleared. In quickack mode, acks are sent immediately rather than delayed.
This flag is not permanent; the TCP stack may switch back to delayed ACK based on internal processing and timeout.
</code>We verified that
wrkonly sets
TCP_NODELAYand does not enable
TCP_QUICKACK:
<code>$ strace -f wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
... setsockopt(52, SOL_TCP, TCP_NODELAY, [1], 4) = 0 ...
</code>Thus the extra 40 ms observed for the latency‑modified Nginx is caused by the client’s delayed ACK behavior.
Conclusion
Use
hping3and
wrkto verify latency for single and concurrent requests.
Use
tracerouteto confirm routing and per‑hop latency.
Capture traffic with
tcpdumpand analyze with Wireshark to ensure packets are transmitted correctly.
Inspect socket options with
straceto detect delayed ACK or missing
TCP_QUICKACKsettings.
Efficient Ops
This public account is maintained by Xiaotianguo and friends, regularly publishing widely-read original technical articles. We focus on operations transformation and accompany you throughout your operations career, growing together happily.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.