Mastering Load Testing: Practical Wrk, GoReplay, and SRE Strategies

Why Automation Testing Falls Behind Product Changes

Many testers spend most of their time on functional testing because the speed of automated test coverage cannot keep up with rapid product iterations, leading to a loss of confidence and a focus on manual functional tests.

Integrating Testing into SRE Practices

SRE engineers should involve motivated testers in reliability work, assigning clear responsibilities to build a dedicated technical‑risk team.

Core Load‑Testing Techniques

Effective load testing ("pressure testing") requires mastering three key areas:

Understanding System Bottlenecks – Use performance analysis tools such as perf, bpftools, and strace to pinpoint why system load spikes. The diagram from Brendan Gregg’s bcc‑tools illustrates common analysis methods.

Capacity Planning for Different Load Scenarios – Establish baseline metrics for promotional events, daily peaks, and troughs to enable accurate comparisons.

Addressing Link‑Length Performance Issues – Code changes can lengthen service call chains, increasing network latency and serialization overhead; shortening these chains is a primary optimization target.

Load‑Testing Blacklist Concept

When evaluating a service’s performance or planning capacity, include both single‑application and full‑chain tests. Services that have never been load‑tested should be blocked from production until they meet defined performance criteria.

Load‑Testing Tool: Wrk

Wrk is an open‑source HTTP benchmark tool that combines multithreading with an event‑notification system (epoll/kqueue), allowing simulation of tens of thousands of concurrent requests while using minimal resources.

It also supports LuaJIT scripts for complex request generation, response handling, and custom reporting. For simple single‑endpoint benchmarks, Wrk is the recommended choice.

Traffic‑Mirroring Tool: GoReplay

GoReplay, written in Go, captures live HTTP traffic at the entry reverse‑proxy and replays it in a test environment. It can scale traffic volume (e.g., amplify requests tenfold) and optionally store captured data in Kafka for further processing.

For users preferring Nginx as the traffic entry point, the built‑in ngx_http_mirror_module (available from Nginx 1.13.4) offers a lightweight mirroring option, though it requires configuration changes on the production server.

Practical Tips for Safe Traffic Replay

Wrap GoReplay commands in scripts instead of invoking gor directly; validate the --output-http target to avoid accidental IP mistakes.

Use --output-file to store replay data on disk, then copy it to an isolated test VPC before replaying.

Integrate traffic capture into change‑control processes and require a secondary approval before execution.

Start replay with GET requests only; enable POST or other methods after confirming stability.

Example GoReplay Command

./gor --input-raw :80 --output-stdout

Running the command displays each incoming HTTP request and its raw payload, as shown in the accompanying screenshots.