How Thread Count, Request Volume, and Latency Shape Performance Test Accuracy

Introduction

The author previously examined various sources of error in performance testing and built a demo to explore how different parameters influence measurement deviation. The demo uses a ThreadLimitTimesCount<String> model and sleep() to simulate request/response cycles.

/**
 * Performance testing deviation demo class
 * by: FunTester
 */
class DeviationTest extends SourceCode {
    static void main(String[] args) {
        def ts = []
        10.times {
            ts << new FunTester(StringUtil.getString(10), 60)
        }
        new Concurrent(ts, "FunTester deviation analysis").start()
    }
    private static class FunTester extends ThreadLimitTimesCount<String> {
        FunTester(String s, int times) {
            super(s, times, null)
        }
        @Override
        protected void doing() throws Exception {
            sleep(0.1)
        }
        @Override
        ThreadBase clone() {
            new FunTester(StringUtil.getString(10), times)
        }
    }
}

Thread Count Impact

Experiments were run with fixed request counts while varying the number of concurrent threads.

{
  "rt":103,
  "failRate":0.0,
  "threads":20,
  "deviation":"3.29%",
  "errorRate":0.0,
  "executeTotal":1000,
  "qps2":187.79,
  "total":1000,
  "qps":194.17,
  "startTime":"2021-04-05 15:40:58",
  "endTime":"2021-04-05 15:41:04",
  "mark":"FunTester deviation analysis 051540"
}

{
  "rt":103,
  "failRate":0.0,
  "threads":40,
  "deviation":"2.7%",
  "errorRate":0.0,
  "executeTotal":2000,
  "qps2":377.86,
  "total":2000,
  "qps":388.35,
  "startTime":"2021-04-05 15:43:19",
  "endTime":"2021-04-05 15:43:24",
  "mark":"FunTester deviation analysis 051543"
}

{
  "rt":102,
  "failRate":0.0,
  "threads":100,
  "deviation":"3.7%",
  "errorRate":0.0,
  "executeTotal":5000,
  "qps2":944.11,
  "total":5000,
  "qps":980.39,
  "startTime":"2021-04-05 15:44:33",
  "endTime":"2021-04-05 15:44:38",
  "mark":"FunTester deviation analysis 051544"
}

{
  "rt":103,
  "failRate":0.0,
  "threads":200,
  "deviation":"3.2%",
  "errorRate":0.0,
  "executeTotal":10000,
  "qps2":1879.70,
  "total":10000,
  "qps":1941.75,
  "startTime":"2021-04-05 15:45:27",
  "endTime":"2021-04-05 15:45:33",
  "mark":"FunTester deviation analysis 051545"
}

The deviation varied only slightly across thread counts, indicating that thread number has a negligible effect on measurement error.

Request Count (Time & Count) Impact

With a fixed thread pool (20 threads), the number of requests was increased.

{
  "rt":105,
  "failRate":0.0,
  "threads":20,
  "deviation":"5.28%",
  "errorRate":0.0,
  "executeTotal":400,
  "qps2":180.42,
  "total":400,
  "qps":190.48,
  "startTime":"2021-04-05 15:48:19",
  "endTime":"2021-04-05 15:48:21",
  "mark":"FunTester deviation analysis 051548"
}

{
  "rt":104,
  "failRate":0.0,
  "threads":20,
  "deviation":"2.79%",
  "errorRate":0.0,
  "executeTotal":1000,
  "qps2":186.95,
  "total":1000,
  "qps":192.31,
  "startTime":"2021-04-05 15:49:42",
  "endTime":"2021-04-05 15:49:48",
  "mark":"FunTester deviation analysis 051549"
}

{
  "rt":103,
  "failRate":0.0,
  "threads":20,
  "deviation":"1.51%",
  "errorRate":0.0,
  "executeTotal":2000,
  "qps2":191.24,
  "total":2000,
  "qps":194.17,
  "startTime":"2021-04-05 15:50:56",
  "endTime":"2021-04-05 15:51:06",
  "mark":"FunTester deviation analysis 051550"
}

{
  "rt":103,
  "failRate":0.0,
  "threads":20,
  "deviation":"0.69%",
  "errorRate":0.0,
  "executeTotal":4000,
  "qps2":192.83,
  "total":4000,
  "qps":194.17,
  "startTime":"2021-04-05 15:52:23",
  "endTime":"2021-04-05 15:52:44",
  "mark":"FunTester deviation analysis 051552"
}

Increasing the number of requests consistently reduced the deviation, confirming that larger sample sizes improve statistical accuracy.

Response‑Time Impact

Fixed 20 threads and 100 requests were used while varying the simulated response latency.

{
  "rt":52,
  "failRate":0.0,
  "threads":20,
  "deviation":"3.99%",
  "errorRate":0.0,
  "executeTotal":2000,
  "qps2":369.28,
  "total":2000,
  "qps":384.62,
  "startTime":"2021-04-05 15:56:36",
  "endTime":"2021-04-05 15:56:42",
  "mark":"FunTester deviation analysis 051556"
}

{
  "rt":103,
  "failRate":0.0,
  "threads":20,
  "deviation":"1.25%",
  "errorRate":0.0,
  "executeTotal":2000,
  "qps2":191.75,
  "total":2000,
  "qps":194.17,
  "startTime":"2021-04-05 15:58:00",
  "endTime":"2021-04-05 15:58:10",
  "mark":"FunTester deviation analysis 051558"
}

{
  "rt":203,
  "failRate":0.0,
  "threads":20,
  "deviation":"0.95%",
  "errorRate":0.0,
  "executeTotal":2000,
  "qps2":97.58,
  "total":2000,
  "qps":98.52,
  "startTime":"2021-04-05 15:59:25",
  "endTime":"2021-04-05 15:59:46",
  "mark":"FunTester deviation analysis 051559"
}

Longer response times yielded lower deviation, matching the intuition that a larger absolute latency reduces the relative impact of measurement noise.

Response‑Time Dispersion Coefficient

A random factor (0,1] was added to the sleep time to simulate variance. Different dispersion levels were tested.

{
  "rt":153,
  "failRate":0.0,
  "threads":20,
  "deviation":"3.56%",
  "errorRate":0.0,
  "executeTotal":2000,
  "qps2":126.07,
  "total":2000,
  "qps":130.72,
  "startTime":"2021-04-05 16:06:34",
  "endTime":"2021-04-05 16:06:50",
  "mark":"FunTester deviation analysis 051606"
}

{
  "rt":202,
  "failRate":0.0,
  "threads":20,
  "deviation":"5.3%",
  "errorRate":0.0,
  "executeTotal":2000,
  "qps2":93.76,
  "total":2000,
  "qps":99.01,
  "startTime":"2021-04-05 16:10:32",
  "endTime":"2021-04-05 16:10:54",
  "mark":"FunTester deviation analysis 051610"
}

{
  "rt":355,
  "failRate":0.0,
  "threads":20,
  "deviation":"4.85%",
  "errorRate":0.0,
  "executeTotal":2000,
  "qps2":53.60,
  "total":2000,
  "qps":56.34,
  "startTime":"2021-04-05 16:13:14",
  "endTime":"2021-04-05 16:13:51",
  "mark":"FunTester deviation analysis 051613"
}

{
  "rt":590,
  "failRate":0.0,
  "threads":20,
  "deviation":"10.51%",
  "errorRate":0.0,
  "executeTotal":2000,
  "qps2":30.34,
  "total":2000,
  "qps":33.90,
  "startTime":"2021-04-05 16:20:41",
  "endTime":"2021-04-05 16:21:47",
  "mark":"FunTester deviation analysis 051620"
}

Higher dispersion (greater randomness) led to larger deviation, confirming that variability in response time amplifies measurement error.

Before & After Timing Methods

The author compared two wrapper methods— before and after —that add extra latency around the request. Longer added latency increased the overall test duration, reduced QPS, and consequently raised the deviation.

{
  "rt":103,
  "failRate":0.0,
  "threads":20,
  "deviation":"9.82%",
  "errorRate":0.0,
  "executeTotal":2000,
  "qps2":175.10,
  "total":2000,
  "qps":194.17,
  "startTime":"2021-04-05 16:32:31",
  "endTime":"2021-04-05 16:32:43",
  "mark":"FunTester deviation analysis 051632"
}

{
  "rt":103,
  "failRate":0.0,
  "threads":20,
  "deviation":"33.29%",
  "errorRate":0.0,
  "executeTotal":2000,
  "qps2":129.53,
  "total":2000,
  "qps":194.17,
  "startTime":"2021-04-05 16:33:10",
  "endTime":"2021-04-05 16:33:25",
  "mark":"FunTester deviation analysis 051633"
}

These results align with the theoretical expectation that longer per‑request overhead reduces throughput and inflates relative error.

Conclusions

Thread count has minimal impact on measurement deviation.

Increasing the number of requests (sample size) significantly lowers error.

Longer response latency reduces relative deviation, while higher latency variance increases it.

Additional wrapper latency (before/after) worsens error by decreasing QPS.

Future posts will explore how exception handling, regex processing, asynchronous completion, and locked resources affect performance‑testing accuracy.