Load Testing and Tuning Insights for a Spring Cloud Microservice System

Test Objective

The performance workflow follows six stages: environment preparation → test script creation → load execution → bottleneck analysis → parameter tuning → re‑verification.

Environment Preparation

Two replicas per microservice.

Container resources: 2 CPU / 4 GB RAM.

Database container: 2 CPU / 4 GB RAM.

Load‑generator machine: 4 CPU / 8 GB RAM.

JMeter Test Plan

Download and start JMeter 5.6.3:

# Download JMeter
wget https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.6.3.tgz
 tar -zxvf apache-jmeter-5.6.3.tgz
# Start GUI
./apache-jmeter-5.6.3/bin/jmeter

Thread group configuration:

200 virtual users.

Ramp‑up time 10 seconds.

50 loops per thread.

HTTP defaults point to http://localhost:8080. The order‑creation request is a POST with JSON payload {"userId":1,"productId":100,"quantity":2,"amount":198}.

CSV Data‑Driven Scenario

# test-data.csv
userId,productId,quantity,amount
1,100,2,198
2,101,1,99
3,102,3,297
4,103,2,198
5,104,1,99

Step‑wise Load Increase (Concurrency Thread Group)

<kg.apc.jmeter.threads.ConcurrencyThreadGroup>
  <intProp name="ThreadGroup.target_level">500</intProp> <!-- target concurrency -->
  <intProp name="ThreadGroup.ramp_up">60</intProp>   <!-- ramp‑up seconds -->
  <intProp name="ThreadGroup.hold_load">120</intProp> <!-- steady load -->
  <intProp name="ThreadGroup.ramp_down">30</intProp>  <!-- ramp‑down -->
</kg.apc.jmeter.threads.ConcurrencyThreadGroup>

Command‑Line Execution

# Non‑GUI mode
./jmeter -n -t test-plan.jmx -l result.jtl -e -o report/
# Options:
# -n: non‑GUI
# -t: test plan file
# -l: result log
# -e: generate report
# -o: output directory

Load Execution and Data Collection

Shell script iterates over six concurrency levels (50, 100, 200, 300, 400, 500) and runs each test for 60 seconds:

# bench-test.sh
#!/bin/bash
CONCURRENCY=(50 100 200 300 400 500)
DURATION=60
for c in "${CONCURRENCY[@]}"; do
  echo "=========================================="
  echo "Load concurrency: $c"
  echo "=========================================="
  jmeter -n -t order-test.jmx \
    -Jthreads=$c \
    -Jrampup=10 \
    -Jduration=$DURATION \
    -l result_${c}.jtl
  curl -s http://localhost:9090/api/v1/query?query=sum(rate(http_server_requests_seconds_count[1m])) | jq '.'
  sleep 30
done

Raw benchmark results (QPS, P99 latency, error rate, CPU, memory) are:

Concurrency 50 – QPS 850, P99 45 ms, error 0 %, CPU 35 %, memory 40 %.

Concurrency 100 – QPS 1 600, P99 68 ms, error 0 %, CPU 55 %, memory 55 %.

Concurrency 200 – QPS 2 850, P99 112 ms, error 0 %, CPU 78 %, memory 70 %.

Concurrency 300 – QPS 3 500, P99 186 ms, error 0.05 %, CPU 92 %, memory 85 %.

Concurrency 400 – QPS 3 800, P99 356 ms, error 2.30 %, CPU 98 %, memory 92 %.

Concurrency 500 – QPS 3 850, P99 689 ms, error 8.50 %, CPU 100 %, memory 95 %.

Performance Bottleneck Analysis

Four typical bottlenecks and the tools used for diagnosis:

CPU saturation – observed when CPU > 90 % and QPS stops growing. Detected with top or kubectl top pod.

Memory pressure / GC – OOM or frequent GC. Examined via jstat and GC logs.

Database connection pool – full pool or slow queries. Investigated with show processlist and the slow‑query log.

Network issues – connection timeouts or retransmissions. Monitored with netstat and tcpdump.

JVM Tuning

# Before tuning
JAVA_OPTS="-Xms1g -Xmx1g"
# After tuning
JAVA_OPTS="-Xms2g -Xmx2g \
  -XX:+UseG1GC \
  -XX:MaxGCPauseMillis=200 \
  -XX:+UseContainerSupport \
  -XX:InitialRAMPercentage=50 \
  -XX:MaxRAMPercentage=75 \
  -XX:+PrintGCDetails \
  -XX:+PrintGCDateStamps \
  -Xloggc:/var/log/gc.log"

Database Connection Pool Tuning (HikariCP)

# Before
spring:
  datasource:
    hikari:
      maximum-pool-size: 10
# After
spring:
  datasource:
    hikari:
      maximum-pool-size: 50
      minimum-idle: 10
      connection-timeout: 30000
      idle-timeout: 600000
      max-lifetime: 1800000
      leak-detection-threshold: 60000

Sentinel Rate‑Limiting Rule

spring:
  cloud:
    sentinel:
      flow:
        - resource: "order:create"
          count: 1000   # QPS threshold
          grade: 1      # QPS type
          controlBehavior: 0 # fast‑fail

Tuning Effects

After applying the above optimizations, the performance improves as follows (QPS / P99 before → after):

Concurrency 50 – QPS 850 → 920, P99 45 ms → 38 ms.

Concurrency 100 – QPS 1 600 → 1 850, P99 68 ms → 52 ms.

Concurrency 200 – QPS 2 850 → 3 400, P99 112 ms → 85 ms.

Concurrency 300 – QPS 3 500 → 4 500, P99 186 ms → 128 ms.

Concurrency 400 – QPS 3 800 → 5 100, P99 356 ms → 198 ms.

Concurrency 500 – QPS 3 850 → 5 300, P99 689 ms → 312 ms.

Tuning Checklist

JVM

Heap size: 1 GB → 2 GB – reduces GC frequency.

GC algorithm: Parallel → G1 – lowers pause times.

Container awareness: disabled → enabled – respects container limits.

Database

Connection pool size: 10 → 50 – supports higher concurrency.

Slow‑query log: disabled → enabled – faster problem identification.

Indexes: missing → added – improves query speed.

Application

Log level: DEBUG → INFO – reduces I/O.

Async processing: none → enabled – moves non‑critical work off the main thread.

Cache: none → multi‑level cache – reduces DB load.