Master JMeter Distributed Load Testing: Setup, Assertions, and Performance Analysis

1. Introduction

For operations engineers, understanding server performance bottlenecks, concurrency limits, and whether horizontal or vertical scaling yields better improvements is essential. This requires both performance testing and monitoring.

2. JMeter Distributed Load Testing Overview

Single‑machine JMeter is limited by memory, CPU, and network I/O; distributed testing uses multiple machines to generate load. The architecture diagram is shown.

Principle: If the JMeter server is set to 10 threads and 100 loops, each agent will generate 10×100 requests; with three agents the total is 3000 requests.

3. Building a Distributed JMeter Environment

3.1. Prerequisites

Server (controller) on Windows, agents on Linux (CentOS 6.5).

Bandwidth of agents must exceed that of the target server.

Server and agents must be on the same network and subnet.

Time synchronization between server and agents.

Firewalls disabled.

3.2. Windows JMeter Deployment

(1) Install JDK and set PATH.

(2) Download the latest binary package.

(3) Extract JMeter, set PATH, and run jmeter to verify.

3.3. Linux JMeter Deployment

(1) Download and unzip JMeter.

wget http://mirrors.tuna.tsinghua.edu.cn/apache//jmeter/binaries/apache-jmeter-3.1.zip
unzip apache-jmeter-3.1.zip -d /usr/local/
cd /usr/local/
ln -s apache-jmeter-3.1/ jmeter

(2) Create a startup script for the JMeter agent.

#!/bin/bash
# chkconfig: 345 26 74
# description: jmeter agent
myip=`ifconfig eth0 |awk '/inet addr/{gsub(/addr:/,"");print $2}'`
cmd="/usr/local/jmeter/bin/jmeter-server -Djava.rmi.server.hostname=$myip"
start(){
  $cmd &
}
stop(){
  jmeter_pid=`ps aux | grep jmeter-server | grep -v grep | awk '{print $2}'`
  for pid in $jmeter_pid; do
    kill -9 $pid
  done
}
act=$1
case $act in
  'start') start;;
  'stop') stop;;
  'restart') stop; sleep 2; start;;
  *) echo '[start|stop|restart]';;
esac

(3) Start the agent and verify it listens on port 1099.

# /etc/init.d/jmeter-agent start
# netstat -lntp | grep 1099
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN 414/java

3.4. Distributed Configuration

Ensure server and agents are installed correctly, agents listen on 1099, edit JMeter’s jmeter.properties to add agent hosts, and start the server with the appropriate IP. jmeter -Djava.rmi.server.hostname=192.168.10.61 Verify the remote host list appears in JMeter and that requests are dispatched to each agent.

4. JMeter Assertions

Two common assertions: response assertion (checks content) and response time assertion (checks latency). Configure them under the HTTP Request sampler.

Example: consider a request successful only if the returned code equals 2000; otherwise fail. Also set a response time threshold of 1 ms.

5. JMeter Variable Configuration

Use variables to simplify changing API endpoints across multiple thread groups. Define variables in the Test Plan and reference them with ${varName}.

6. Performance Test Result Analysis

Aggregated report shows a performance knee at 27 k requests per second, with a peak TPS of 6 k, 99 % response time 60 ms, max 71 ms. Concurrency bottleneck appears around 6 k TPS.

Response time remains stable up to 3.5 k TPS.

Note: analysis should focus on hotspot interfaces.

7. Monitoring During Tests

7.1. Concurrency Test Monitoring

For high‑volume tests (e.g., 100 k requests), traditional tools like Zabbix may be too coarse; use real‑time commands such as dstat or glances to capture CPU, memory, and I/O.

7.2. Stability Test Monitoring

Long‑running tests (e.g., 4 k requests per second for 12 hours) benefit from Zabbix to track metrics like Tomcat request rate, network traffic, HTTP status codes, and resource usage.