Comparative Analysis of Six Server Load‑Testing Tools: LoadRunner, JMeter, Spirent Avalanche, Siege, Tsung, and Locust

The 360QA performance team analyzes six server‑side pressure‑testing tools (LoadRunner, JMeter, Spirent Avalanche, Siege, Tsung, and Locust) based on project practice.

Tool characteristics comparison

Scenario 1 : HTTP/JSON API with 100 concurrent users and 500 TPS. Transaction flow includes obtaining a session ID, logging in to get a token, searching for items, and processing each item’s result with retry logic for "wait" states.

Siege only supports simple protocol interactions and is unsuitable.

Tsung’s XML script language cannot express required programming logic (e.g., breaking loops, dynamic list reconstruction) without Erlang extensions, making it a poor choice.

LoadRunner, JMeter, and Locust use C, Java, and Python respectively, allowing easy implementation of the needed logic.

Scenario 2 : RADIUS EAP‑MD5 authentication for 100 NAS devices at 2000 QPS.

Preferred tool is Avalanche, which fully supports EAP‑MD5 and exceeds performance needs.

LoadRunner can be extended via C/C++ dynamic libraries using FreeRADIUS (Linux only).

JMeter can be extended with the Java‑based JRadius library.

Tsung lacks an EAP‑MD5 implementation in Erlang and is excluded.

Locust can use the Python pyrad library, but performance is significantly lower than C/Java versions, so it is also excluded.

Ultimately, LoadRunner and JMeter satisfy the requirements, with the final choice depending on C/Java expertise.

Scenario 3 : Simple HTTP GET/POST interface requiring verification of HTTP 200 responses, targeting 500 k concurrent users and 1.5 M QPS in a clustered environment.

All tools can meet the functional need.

Considering resource cost, Avalanche and Siege are preferred for their low resource consumption and high performance.

Avalanche can deliver 180 k QPS per 1 Gbps port; nine ports achieve the 1.5 M QPS goal.

Siege can reach 180 k QPS on a dual‑socket E5 server; nine such servers are needed, whereas Tsung would require ~25 servers and LoadRunner ~70 servers.

Siege lacks native distributed execution, so a simple wrapper (e.g., Python pexpect or shell rsh) is required for multi‑machine runs.

Scenario 4 : Mixed‑traffic performance testing for an NGFW product (NSS Labs Real‑world test).

Avalanche: commercial hardware, supports multi‑protocol client/server simulation, controllable traffic mix, detailed reporting – recommended.

Siege: does not support mixed‑protocol traffic.

LoadRunner, JMeter, Tsung, Locust: require building separate protocol servers and complex script configuration; mixed‑traffic control is difficult.

The article then outlines the step‑by‑step Avalanche configuration for HTTP traffic, including client GET actions and server upload of a 112 KB file, followed by association setup and traffic verification.

Scenario 5 : Java JMS/RMI interface with complex business logic, targeting 100 concurrent users and 3000 TPS.

Only LoadRunner and JMeter support Java‑based RPC interfaces; LoadRunner loads a JVM, while JMeter is a native Java implementation.

Key factors for tool selection :

Business logic of the system under test

Expected load magnitude

Tool extensibility for custom development

Team’s proficiency with the tool’s scripting language

Summary of the six tools:

Avalanche – best for gateway‑type products, high‑scale multi‑protocol tests, but not for complex business logic.

Siege – ideal for HTTP/HTTPS extreme benchmark with low cost, unsuitable for complex interactions.

LoadRunner – versatile for most scenarios, extensive documentation and community support.

JMeter – versatile, especially for Java‑specific interfaces (RMI/JMS).

Tsung – low‑cost high‑scale option, limited for complex logic or extensibility.

Locust – flexible for extensible scenarios, less suited for ultra‑high‑scale low‑cost needs.

Choosing the right tool can greatly improve testing efficiency; the article hopes to assist readers in performance‑testing projects.