Implementing Precise QPS Metrics and Deviation Tracking in a Performance Test Framework

Background and Goal

In the previous two articles the author analyzed the sources of error in performance testing from a textual perspective. The next step is quantitative analysis, which requires the testing framework to support statistical collection of these errors.

QPS Calculation Formulas

Two formulas for QPS (queries per second) are introduced:

QPS = total request count / total time
QPS = count(r)/T

QPS = thread count / average response time
QPS = thread/rt

The author prefers the first formula for implementation.

Extending the Data Model

Two new properties are added to the PerformanceResultBean class:

/**
 * QPS calculated by QPS = count(r)/T; in fixed‑QPS mode this value comes from the preset QPS
 */
double qps2;

/**
 * Theoretical deviation for the two statistical modes
 */
String deviation;

In the constructor the values are assigned as follows:

this.qps2 = qps2;
this.deviation = com.funtester.frame.SourceCode.getPercent(
    Math.abs(qps - qps2) * 100 / Math.max(qps, qps2)
);

Implementation in Fixed‑Thread Model

The ThreadBase class gains an executeNum field to record the number of executions, which usually matches the number of response‑time records.

/**
 * Execution count, generally equal to the number of response‑time records
 */
public int executeNum;

At the end of a test run, statistics from all threads are aggregated:

threads.forEach(x -> {
    if (x.status()) failTotal++;
    errorTotal += x.errorNum;
    executeTotal += x.executeNum;
});

The QPS2 value is then calculated:

double qps2 = (executeTotal + errorTotal) * 1000.0 / (endTime - startTime);

Implementation in Fixed‑QPS Model

In the fixed‑QPS scenario the total request count is already tracked by FixedQpsConcurrent:

public static AtomicInteger executeTimes = new AtomicInteger(0);

The run method of a subclass increments counters and records timing information:

@Override
public void run() {
    try {
        before();
        threadmark = this.mark == null ? EMPTY : this.mark.mark(this);
        long s = Time.getTimeStamp();
        doing();
        long e = Time.getTimeStamp();
        // Increment counter
        FixedQpsConcurrent.executeTimes.getAndIncrement();
        int diff = (int) (e - s);
        FixedQpsConcurrent.allTimes.add(diff);
        if (diff > HttpClientConstant.MAX_ACCEPT_TIME)
            FixedQpsConcurrent.marks.add(diff + CONNECTOR + threadmark + CONNECTOR + Time.getNow());
    } catch (Exception e) {
        FixedQpsConcurrent.errorTimes.getAndIncrement();
        logger.warn("Execution task failed! Mark:{}", threadmark, e);
    } finally {
        after();
    }
}

Because the second QPS formula does not apply here, the expected QPS value is used directly:

int qps2 = baseThread.qps; // use preset QPS instead of calculated qps2

Conclusion and Next Steps

The core work of adding statistical support is completed. The author plans to perform quantitative error‑comparison experiments across different scenarios in upcoming articles.