Essential Best Practices for Accurate HTTP Load Testing

Many people discuss HTTP server performance testing, but repeated mistakes often undermine the credibility of results. After extensive experience with high‑performance proxy caches and origin server benchmarks, I share the most important considerations.

Consistency

Always test at the same point in time; any OS upgrade or other bandwidth‑consuming process can affect results, so keep the test environment fixed.

Running tests inside a VM adds an extra abstraction layer and extra processes on the host, which can skew results. Ideally use dedicated hardware, or at least keep all tests within the same session.

1. Separate Machines for Load Generator and Server

Do not place the load generator and the server on the same machine, as the generator will steal resources and the amount stolen varies with server load.

Assign different hardware to the server and the load generator and place them on a closed network. High‑end hardware is unnecessary; consistency is the key.

2. Check Network Capacity

Before testing, determine your network’s capacity to know whether the server or the network is the bottleneck.

Example using iperf:

qa1:~> iperf -c qa2
--------------------------------------------
Client connecting to qa2, TCP port 5001
TCP window size: 16.0 KByte (default)
--------------------------------------------
[  3] local 192.168.1.106 port 56014 connected with 192.168.1.107 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.10 GBytes   943 Mbits/sec

The output shows a 943 Mbps throughput on a gigabit link (the shortfall is due to TCP overhead).

Record current bandwidth usage, e.g., with httperf: Net I/O: 23399.7 KB/s (191.7*10^6 bps) In this example only 192 Mbps is used.

Avoid saturating the network; keep test traffic below about two‑thirds of the available bandwidth and monitor for packet loss on cheap NICs or switches.

3. Remove OS Limits

Adjust TCP parameters and ensure the server does not run out of file descriptors.

4. Avoid Load‑Generator Bottlenecks

Make sure the load‑generator hardware exceeds the server’s capabilities; otherwise the generator becomes the limiting factor.

Watch for client‑side issues such as exhausted ephemeral ports and tune TIME_WAIT or use HTTP keep‑alive.

Example of httperf error summary:

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

5. Do Not Mistake Overload Capacity for True Capacity

Gradually increase load until performance degrades, then plot the peak‑then‑decline curve to understand overload behavior.

6. Test Long Enough

Because buffers in the application, OS, and network need to stabilize, tests shorter than 30 seconds are unreliable; aim for at least 3 minutes, preferably 5–10 minutes for published data.

7. Use Realistic Payloads

Testing with a 4‑byte response is meaningless; use 4 KB or 100 KB payloads to reflect real workloads, and also test with many idle connections (e.g., 10 000).

8. Report More Than Averages

Average response time hides latency spikes; provide timelines, histograms, and statistics such as min, max, and standard deviation.

Example httperf output:

Total: connections 180000 requests 180000 replies 180000 test-duration 179.901 s
Connection rate: 1000.0 conn/s (99.9 ms/conn, <=2 concurrent connections)
Connection time [ms]: min 0.4 avg 0.5 max 12.9 median 0.5 stddev 0.4
Connection time [ms]: connect 0.1
Connection length [replies/conn]: 1.000
Request rate: 1000.0 req/s (.9 ms/req)
Request size [B]: 79.0
Reply rate [replies/s]: min 999.1 avg 1000.0 max 1000.2 stddev 0.1 (35 samples)
Reply time [ms]: response 0.4 transfer 0.0
Reply size [B]: header 385.0 content 1176.0 footer 0.0 (total 1561.0)
Reply status: 1xx=0 2xx=0 3xx=0 4xx=1800 5xx=0

9. Publish Full Context

Provide all necessary information—hardware, OS version and configuration, network settings, server and load‑generator versions, load details, and even source code—so others can reproduce the results, ideally via a public repository.

10. Try Different Tools

Relying on a single tool can bias results; different tools (e.g., httperf, autobench, siege) have varying capabilities and may interact differently with servers.