How to Efficiently Process Test Data in a Multithreaded Performance Engine

In the performance testing engine we designed, test data handling occurs in two places: the multithreaded task class and the multithreaded executor class.

Design Options

Report data from each task to the executor after the task finishes.

Let the executor collect data from all task objects after every task has completed.

The main difference is where the reporting/collection logic resides. From a class‑design perspective, placing it in the executor keeps the task class simple, allowing developers to focus on task‑specific extensions without worrying about data handling. Reporting from the task class adds overhead to the task execution time and may affect test accuracy.

Implementation in start()

We add data aggregation to the executor’s start() method:

for (int i = 0; i < tasks.size(); i++) {
    ThreadTask threadTask = tasks.get(i);
    this.executeNumStatistic.addAndGet(threadTask.executeNum); // accumulate execution count
    this.errorNumStatistic.addAndGet(threadTask.errorNum);   // accumulate error count
    this.costTimeStatistic.addAll(threadTask.costTime);      // collect request times
}

After the loop we compute the total expected executions and the test duration:

int sum = tasks.stream().mapToInt(f -> f.totalNum).sum();
long costTime = (endTimestamp - startTimestamp) / 1000; // test duration in seconds
System.out.println(String.format(
    "Task finished! Duration: %d seconds, Expected executions: %d, Actual executions: %d, Errors: %d, Collected times: %d",
    costTime, sum, executeNumStatistic.get(), errorNumStatistic.get(), costTimeStatistic.size()));

How to Handle Massive costTimeStatistic Data

Best option: Do not collect locally; stream data to a dedicated service for real‑time monitoring and later reporting.

Middle option: Collect locally and hand off to a third‑party tool (e.g., Python tabulation libraries) for processing.

Worst option: Collect and process locally on the load‑generator machine, which can degrade performance due to Java collection overhead and is generally discouraged.

When a monitoring system is unavailable, a lightweight local analysis can still be useful. Below is a simple method that computes minimum, maximum, average, and several percentiles (p50, p90, p95, p99, p999) from the collected times.

/**
 * Statistic data
 */
public static void statisticData(List<Integer> data) {
    Collections.sort(data);
    int min = data.get(0);
    int max = data.get(data.size() - 1);
    int average = (int) data.stream().mapToInt(x -> x).average().getAsDouble();
    int p50 = data.get((int) (0.5 * data.size()));
    int p90 = data.get((int) (0.9 * data.size()));
    int p95 = data.get((int) (0.95 * data.size()));
    int p99 = data.get((int) (0.99 * data.size()));
    int p999 = data.get((int) (0.999 * data.size()));
    System.out.println("Min: " + min);
    System.out.println("Max: " + max);
    System.out.println("Average: " + average);
    System.out.println("p50: " + p50);
    System.out.println("p90: " + p90);
    System.out.println("p95: " + p95);
    System.out.println("p99: " + p99);
    System.out.println("p999: " + p999);
}

We then invoke this method after the test finishes:

DataHandleTool.statisticData(costTimeStatistic);

TPS Calculation

Two common ways to compute TPS (transactions per second) are:

Total execution count divided by total test time.

Number of threads divided by average response time.

In ideal conditions both formulas yield the same TPS, but in real tests they differ because the first method includes setup/teardown time, while the second method may over‑estimate TPS when the code outside test() is non‑trivial.

For small teams lacking a monitoring system, we output both TPS values and the detailed statistics at the end of the run.

Refactoring DataHandleTool

We introduce an inner class Quantile to hold all statistical fields and a print() method:

public static class Quantile {
    public int min;
    public int max;
    public int average;
    public int p50;
    public int p90;
    public int p95;
    public int p99;
    public int p999;
    public Quantile(int min, int max, int average, int p50, int p90, int p95, int p99, int p999) {
        this.min = min; this.max = max; this.average = average;
        this.p50 = p50; this.p90 = p90; this.p95 = p95; this.p99 = p99; this.p999 = p999;
    }
    public void print() {
        System.out.println("Min: " + min);
        System.out.println("Max: " + max);
        System.out.println("Average: " + average);
        System.out.println("p50: " + p50);
        System.out.println("p90: " + p90);
        System.out.println("p95: " + p95);
        System.out.println("p99: " + p99);
        System.out.println("p999: " + p999);
    }
}

The statisticData method now returns a Quantile instance:

public static Quantile statisticData(List<Integer> data) {
    Collections.sort(data);
    int min = data.get(0);
    int max = data.get(data.size() - 1);
    int average = (int) data.stream().mapToInt(x -> x).average().getAsDouble();
    int p50 = data.get((int) (0.5 * data.size()));
    int p90 = data.get((int) (0.9 * data.size()));
    int p95 = data.get((int) (0.95 * data.size()));
    int p99 = data.get((int) (0.99 * data.size()));
    int p999 = data.get((int) (0.999 * data.size()));
    return new Quantile(min, max, average, p50, p90, p95, p99, p999);
}

Updated Data Handling in start()

public void handleData() {
    for (int i = 0; i < tasks.size(); i++) {
        ThreadTask threadTask = tasks.get(i);
        this.executeNumStatistic.addAndGet(threadTask.executeNum);
        this.errorNumStatistic.addAndGet(threadTask.errorNum);
        this.costTimeStatistic.addAll(threadTask.costTime);
    }
    int sum = tasks.stream().mapToInt(f -> f.totalNum).sum();
    long costTime = (endTimestamp - startTimestamp) / 1000;
    DataHandleTool.Quantile quantile = DataHandleTool.statisticData(costTimeStatistic);
    System.out.println(String.format("Test TPS: %d, Avg latency: %f", this.tasks.size() * 1000 / quantile.average, quantile.average));
    System.out.println(String.format("Test TPS: %d, Total executions: %d", executeNumStatistic.get() / costTime, executeNumStatistic.get()));
    quantile.print();
    System.out.println(String.format("Task finished! Duration: %d seconds, Expected: %d, Actual: %d, Errors: %d, Collected times: %d", costTime, sum, executeNumStatistic.get(), errorNumStatistic.get(), costTimeStatistic.size()));
}

Finally, the after() method logs a concise summary:

public void after() {
    System.out.println(String.format("Task finished! Expected: %d, Actual: %d, Errors: %d, Collected times: %d", totalNum, executeNum, errorNum, costTime.size()));
}

These changes make the data‑processing flow clearer, enable accurate TPS reporting, and keep the task class lightweight for future extensions.