How to Diagnose and Reduce Linux Network Latency with hping3, wrk, and Wireshark

Background and Problem Statement

Network latency is a critical performance metric for any service. It can be caused by slow transmission, inefficient kernel packet processing, or slow application handling. When a DDoS attack or heavy traffic reaches a Linux server, latency often spikes even if the application layer is optimized.

Key Concepts

Network Latency (RTT) measures the round‑trip time of a packet from source to destination and back. Application Latency adds the time the application spends processing the request, effectively RTT plus processing time.

Tools for Measuring Latency

ping

– ICMP‑based RTT measurement (often disabled by firewalls). traceroute – Shows per‑hop latency using TCP/UDP probes. hping3 – Flexible packet generator; can send TCP SYN probes and report RTT. wrk – HTTP load generator that reports latency distribution under concurrency. tcpdump and Wireshark – Capture and analyze packets for deeper inspection.

Example Environment

Two hosts are used:

host1 (192.168.0.30) – runs two Docker containers: the official Nginx (port 80) and a latency‑modified Nginx (port 8080).

host2 (192.168.0.2) – acts as the client and runs the measurement tools.

Launching the containers on host1

# Official nginx
$ docker run --network=host --name=good -itd nginx
# Latency‑modified nginx (feisky/nginx:latency)
$ docker run --name nginx --network=host -itd feisky/nginx:latency

Verify both containers serve traffic:

$ curl http://127.0.0.1
$ curl http://127.0.0.1:8080

Measuring single‑request latency with hping3

# Port 80
$ hping3 -c 3 -S -p 80 192.168.0.30
# Port 8080
$ hping3 -c 3 -S -p 8080 192.168.0.30

Both ports show ~7 ms RTT for three SYN packets.

Measuring concurrent latency with wrk

# 100 connections, 2 threads, 10 s test on port 80
$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30/
# Same test on port 8080
$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/

Results:

Port 80 (official Nginx) – average latency ≈ 9 ms, 90 % of requests < 9 ms.

Port 8080 (latency Nginx) – average latency ≈ 44 ms, 50 % of requests already > 44 ms.

Deep Dive with Packet Capture

On host1 capture traffic for the delayed port: $ tcpdump -nn tcp port 8080 -w nginx.pcap After running the wrk test from host2, stop tcpdump and open nginx.pcap in Wireshark.

Wireshark automatically applies a filter like tcp.stream eq 24, allowing inspection of the three‑way handshake and subsequent HTTP requests.

The second HTTP request shows a 40 ms gap before the ACK is sent – this is the TCP delayed‑ACK timeout.

Analysis of the Delay

Linux’s default TCP behavior uses delayed acknowledgments, waiting up to ~40 ms for possible additional data before sending an ACK. The wrk client does not enable TCP_QUICKACK, so the delayed ACK mechanism remains active, causing the observed extra latency on the latency‑modified Nginx.

Confirming the socket options with strace shows that wrk sets TCP_NODELAY but not TCP_QUICKACK:

$ strace -f wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
... setsockopt(52, SOL_TCP, TCP_NODELAY, [1], 4) = 0 ...

Conclusion

To troubleshoot increased network latency on Linux:

Use hping3 and wrk to compare single‑request and concurrent latency.

Employ traceroute to verify routing and per‑hop delays.

Capture traffic with tcpdump and analyze with Wireshark to spot protocol‑level issues such as delayed ACKs.

Inspect socket options (e.g., TCP_QUICKACK, TCP_NODELAY) with strace when needed.

Understanding both network‑level and application‑level contributors helps reduce latency and improve user experience.