How Huolala Built a Scalable Distributed Load‑Testing Platform with JMeter

Background and Challenges

In recent years, Huolala's user base and freight order volume have grown rapidly, making system stability increasingly critical. Traditional single‑machine performance testing tools struggle with efficiency, collaboration, and data persistence, prompting the need for a high‑performance testing platform.

Solution and Goals

After evaluating open‑source options, the team chose JMeter as the core engine and extended it for script maintenance, collaborative workflows, and scenario‑based testing, ensuring fast dynamic scaling and reduced communication overhead.

Capability Building

The platform's capabilities are illustrated through key functional components, including module planning and code structure.

├── Dockerfile
├── README.md
├── pom.xml
├── qapt-agent                // test agent service
├── qapt-agent-base           // agent base classes
├── qapt-agent-plugin         // agent & JMeter plugins
├── qapt-api                  // management API
├── qapt-collector            // data collector
├── qapt-hsmart               // script handling
├── qapt-monitor              // data monitoring
└── springboot-module         // base modules
    ├── pom.xml
    ├── springboot-base-boot   // core utilities
    ├── springboot-base-msg    // message management
    ├── springboot-mongodb-auth// permission management
    ├── springboot-mongodb-file// file management
    └── springboot-mongodb-quartz// scheduled tasks

3.1 Common Modules

3.1.1 JMeter Configuration Component

To run JMeter inside a Spring container, configuration files must be placed in a designated location and the JMeter home path set accordingly.

@Configuration
@Slf4j
public class JmeterFileConfig {
    static {
        String jmeterHome = FilesUtil.getJarPath(SpringbootMongodbFileApplication.class);
        log.info("================Jmeter Info====================,JmeterHome:{}", jmeterHome);
        // Resolve path differences between compile and deployment
        if (jmeterHome.endsWith("lib")) {
            jmeterHome = FilesUtil.getParentPath(jmeterHome);
        } else if (jmeterHome.endsWith("qapt-api")) {
            jmeterHome = jmeterHome + "/src/main/resources";
        }
        String jmeterProperties = jmeterHome + File.separator + "bin" + File.separator + "jmeter.properties";
        JMeterUtils.setJMeterHome(jmeterHome);
        JMeterUtils.getProperties(jmeterProperties);
        String localScriptHome = jmeterHome + File.separator + "scripts";
        JMeterUtils.setProperty("LocalScriptHome", localScriptHome);
        JMeterUtils.setProperty("search_paths", jmeterHome + File.separator + "lib");
        JMeterUtils.setProperty("jmeter.home", jmeterHome);
        FilesUtil.addPath(JMeterUtils.getProperty("search_paths"));
        log.info("================Jmeter Bean Started=====================");
    }
}

3.1.2 Unified Log Handling

The system filters logs, adds tracing information, and forwards error logs to reporting queues.

/*
Log listener queue, registers file listener and sends log messages.
*/
@Component
public class LoggerDisruptorQueue {
    private static final Pattern LOG_PATTERN = Pattern.compile("^(\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d{3})\\s+(\\w+)\\s+([\\w\\.]+):\\s+(.*)$");
    private static RingBuffer<LoggerEvent> ringBuffer;
    @Autowired
    LoggerDisruptorQueue(LoggerEventHandler eventHandler) {
        ThreadFactory threadFactory = Executors.defaultThreadFactory();
        LoggerEventFactory factory = new LoggerEventFactory();
        int bufferSize = 2 * 1024;
        Disruptor<LoggerEvent> disruptor = new Disruptor<>(factory, bufferSize, threadFactory);
        disruptor.handleEventsWith(eventHandler);
        ringBuffer = disruptor.getRingBuffer();
        disruptor.start();
    }
    public static void publishEvent(LoggerMessage log) {
        long sequence = ringBuffer.next();
        try {
            LoggerEvent event = ringBuffer.get(sequence);
            event.setLog(log);
        } finally {
            ringBuffer.publish(sequence);
        }
    }
    public static void publishEvent(String reportId, String logLine) {
        Matcher matcher = LOG_PATTERN.matcher(logLine);
        if (matcher.matches()) {
            String timestamp = matcher.group(1);
            String level = matcher.group(2);
            String className = matcher.group(3);
            String message = matcher.group(4);
            LoggerMessage tempLoggerMessage = new LoggerMessage(message, timestamp, "", className, level, reportId);
            long sequence = ringBuffer.next();
            try {
                LoggerEvent event = ringBuffer.get(sequence);
                event.setLog(tempLoggerMessage);
            } finally {
                ringBuffer.publish(sequence);
            }
        }
    }
}

3.2 Load‑Test Agent Management

3.2.1 JMeter Engine

Initially the JMeter engine was launched inside the Spring container, but scalability and class‑path conflicts led to a hybrid approach where script preprocessing occurs in‑process and actual load generation runs via native JMeter command line.

/* Native JMeter startup */
@Slf4j
public class OriginalJmeterService {
    private static String javaHome = SystemUtil.getJavaRuntimeInfo().getHomeDir();
    private static String jmeterHome = FilesUtil.getJarPath(YiapiAgentServerApplication.class);
    private static String classpath;
    private static String jarPath;
    private static ExecutorService executorService = ThreadUtil.newExecutor();
    private ScheduledExecutorService exec = Executors.newScheduledThreadPool(1);
    private ScheduledFuture<?> scheduledFuture;
    private long lastTimeFileSize = 0; // previous file size
    private static String scriptPath = jmeterHome + File.separator + "scripts";
    private static Process process;
    static {
        if (jmeterHome.endsWith("lib")) {
            jmeterHome = FilesUtil.getParentPath(jmeterHome);
        } else if (jmeterHome.endsWith("classes")) {
            jmeterHome = FilesUtil.getParentPath(FilesUtil.getParentPath(jmeterHome)) + "/target/dest";
        }
        classpath = jmeterHome + "/jmeter/lib/*";
        jarPath = jmeterHome + "/jmeter/bin/ApacheJMeter.jar";
    }
    public String run(String scriptPath, ScriptDTO scriptDTO) throws IOException {
        String reportId = scriptDTO.getReportId();
        Report report = scriptDTO.getReport();
        String logPath = FilesUtil.getParentPath(scriptPath) + File.separator + "jmeter.log";
        log.info("FilesUtil.getParentPath:{}", logPath);
        String cmd = javaHome + "/bin/java -Xms4G -Xmx4G -jar " + getSystemProperties(scriptDTO) + jarPath + " -n -t " + scriptPath + " -j " + logPath;
        log.info("run original jmeter and the cmd is:{}", cmd);
        AgentInfo.setAgentStatus(Agent.AGENT_BUSYING);
        File logFile = FileUtil.newFile(logPath);
        executorService.execute(() -> process = RuntimeUtils.exec("output.log", cmd));
        if (report.getType() == Report.REPORT_TYPE_DEBUG) {
            if (scheduledFuture != null) {
                scheduledFuture.cancel(true);
            }
            scheduledFuture = exec.scheduleWithFixedDelay(() -> {
                final RandomAccessFile randomFile;
                try {
                    randomFile = new RandomAccessFile(logFile, "rw");
                } catch (FileNotFoundException e) {
                    throw new RuntimeException(e);
                }
                try {
                    randomFile.seek(lastTimeFileSize);
                    String tmp;
                    while ((tmp = randomFile.readLine()) != null) {
                        String text = new String(tmp.getBytes(StandardCharsets.UTF_8));
                        LoggerDisruptorQueue.publishEvent(reportId, text);
                    }
                    lastTimeFileSize = randomFile.length();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }, 0, 1, TimeUnit.SECONDS);
        }
        return executorService.toString();
    }
}

3.2.2 Agent Deployment

The platform initially used ECS clusters; when scaling to ~200 agents, costs rose, so it switched to serverless containers (ASK) that can provision 500 agents within a minute, with automatic release after tasks complete.

Resources are also managed per project, user, and shared/private dimensions to improve utilization.

3.2.3 Test File Processing

During a test, scripts, data files, and dependent JARs are distributed to target agents, with isolation to avoid cross‑project conflicts.

@Slf4j
@Service
public class JmeterService {
    @Autowired
    private JMeterEngines jMeterEngine;
    @Autowired
    private LoggerEventHandler loggerEventHandler;
    private boolean hasEnvironmentalRisk(HashTree tree, ScriptDTO scriptDTO) {
        // Check ConfigTestElement for environment mismatches
        SearchByClass<ConfigTestElement> configTestElementListeners = new SearchByClass<>(ConfigTestElement.class);
        tree.traverse(configTestElementListeners);
        Collection<ConfigTestElement> configTestElements = configTestElementListeners.getSearchResults();
        for (ConfigTestElement configTestElement : configTestElements) {
            if (configTestElement.isEnabled()) {
                String domain = configTestElement.getPropertyAsString("HTTPSampler.domain");
                if (!StringUtils.isEmpty(domain)) {
                    String env = scriptDTO.getReport().getEnv();
                    if (!hasDomainContainEnv(domain, env)) {
                        sendStopReportErrorMsg(scriptDTO, domain, configTestElement.getName(), env);
                        return true;
                    }
                }
            }
        }
        // Check HTTPSamplerProxy for environment mismatches
        SearchByClass<HTTPSamplerProxy> httpSamplerProxyListeners = new SearchByClass<>(HTTPSamplerProxy.class);
        tree.traverse(httpSamplerProxyListeners);
        Collection<HTTPSamplerProxy> httpSamplerProxies = httpSamplerProxyListeners.getSearchResults();
        for (HTTPSamplerProxy http : httpSamplerProxies) {
            String domain = http.getPropertyAsString("HTTPSampler.domain");
            String env = scriptDTO.getReport().getEnv();
            if (!StringUtils.isEmpty(domain) && !hasDomainContainEnv(domain, env)) {
                sendStopReportErrorMsg(scriptDTO, domain, http.getName(), env);
                return true;
            }
        }
        return false;
    }
}

3.2.4 Plugin Management

Supports protocols such as MQTT and gRPC, as well as internal custom protocols, packaging them for GUI use and agent dependency injection.

3.3 Management Backend

The web UI interacts directly with the backend to manage scripts, scenarios, and agents, while also providing monitoring, batch operations, and automation features.

3.3.1 Script Management

Uploaded JMeter scripts are pre‑processed and stored, enabling per‑script testing.

3.3.2 Scenario Management

Multiple scripts can be combined into scenarios with configurable concurrency for large‑scale full‑link testing.

3.3.3 Communication Protocol

Initially using WebSocket for agent heartbeat and log upload, the platform added HTTP communication to improve scalability as load increased.

@ServerEndpoint("/ws/{cid}/route/{rid}")
@Component
@Slf4j
@Data
public class WebSocketServer {
    private static Map<String, WebSocketService> webSocketServiceMap;
    @OnOpen
    public void onOpen(Session session, @PathParam("cid") String cid, @PathParam("rid") String rid) {
        WsSessionManager.add(cid, session);
        log.info("New client connected, clientId:{}, routeId:{}", cid, rid);
    }
    @OnClose
    public void onClose(@PathParam("cid") String cid, @PathParam("rid") String rid) {
        WsSessionManager.removeAndClose(cid);
        WsSessionManager.removeAndClose(rid);
        AgentServiceImpl agentService = SpringContextUtils.getBean(AgentServiceImpl.class);
        if (!cid.startsWith(WsSessionManager.WEB_CLIENT)) {
            agentService.setOffOnline(cid);
        }
    }
    @OnMessage
    public void onMessage(@PathParam("cid") String cid, String message) {
        if (webSocketServiceMap == null) {
            webSocketServiceMap = SpringContextUtils.getBeans(WebSocketService.class);
        }
        log.debug("session:{}, message:{}", cid, message);
        WsMsg msg;
        try {
            msg = JSON.parseObject(message).toJavaObject(WsMsg.class);
            String uuid = msg.getUuid();
            if (WsSessionManager.futureCache.asMap().containsKey(uuid)) {
                SyncFuture syncFuture = WsSessionManager.futureCache.get(uuid);
                syncFuture.setResponse(message);
            }
        } catch (JSONException e) {
            log.error(e.toString());
            return;
        } catch (ExecutionException e) {
            throw new RuntimeException(e);
        }
        try {
            if (webSocketServiceMap.containsKey(msg.getAction())) {
                webSocketServiceMap.get(msg.getAction()).execute(msg, cid);
            } else {
                log.error("No handler for action:{}", msg.getAction());
            }
        } catch (Exception e) {
            log.error("Handler exception for action:{}", msg.getAction());
            e.printStackTrace();
        }
    }
    @OnError
    public void onError(@PathParam("cid") String cid, @PathParam("rid") String rid, Throwable e) {
        WsSessionManager.removeAndClose(cid);
        WsSessionManager.removeAndClose(rid);
        log.error("WebSocket error, cid:{}, rid:{}, reason:{}", cid, rid, e.getMessage());
    }
    // Additional sendMessage utilities omitted for brevity
}

3.4 Collector

3.4.1 Single‑Agent Data

Agents aggregate their own metrics and send a summary to Kafka once per second to avoid overwhelming the message pipeline.

// Collect and send aggregated data from each agent to Kafka
@Slf4j
public class HllBackendListenerClient extends AbstractBackendListenerClient {
    @Override
    public void setupTest(BackendListenerContext ctx) throws Exception {
        // setup logic
    }
    @Override
    public void handleSampleResults(List<SampleResult> list, BackendListenerContext ctx) {
        // process and send data
    }
    @Override
    public void teardownTest(BackendListenerContext ctx) throws Exception {
        analysis();
        schedule.shutdown();
    }
    private void analysis() {
        // analysis logic
    }
}

3.4.2 Aggregate Data

The collector groups incoming SampleState objects by label, aggregates performance records, persists them, and logs processing statistics.

@Slf4j
public class AnalysisProcessor implements Processor<String, String> {
    private ProcessorContext context;
    private AtomicInteger processorNumber = new AtomicInteger();
    private PerformanceRecordRepository performanceRecordRepository = SpringContextUtils.getBean(PerformanceRecordRepository.class);
    private ReportRepository reportRepository = SpringContextUtils.getBean(ReportRepository.class);
    private ConcurrentHashMap<String, List<SampleState>> sampleMap = new ConcurrentHashMap<>();
    @Override
    public void init(ProcessorContext context) {
        this.context = context;
        processorNumber.getAndIncrement();
        if (processorNumber.get() <= 1) {
            this.context.schedule(Report.REPORT_INTERVAL, PunctuationType.WALL_CLOCK_TIME, ts -> analysis());
            log.info("init AnalysisProcessor");
        }
    }
    @Override
    public void process(String key, String message) {
        SampleState sampleState = JSON.parseObject(message, SampleState.class);
        sampleMap.computeIfAbsent(key, k -> new ArrayList<>()).add(sampleState);
    }
    @Override
    public void close() {}
    private void analysis() {
        sampleMap.forEach((reportId, sampleStates) -> {
            if (sampleStates.isEmpty()) {
                sampleMap.remove(reportId);
                return;
            }
            long start = System.currentTimeMillis();
            Collector<SampleState, PerformanceRecordState, PerformanceRecord> c = Collector.of(
                PerformanceRecordState::new,
                PerformanceRecordState::accumulator,
                PerformanceRecordState::combiner,
                PerformanceRecordState::finisher);
            Map<String, PerformanceRecord> reportRecord = sampleStates.stream()
                .collect(Collectors.groupingBy(SampleState::getLabel, c));
            AtomicInteger i = new AtomicInteger();
            reportRecord.forEach((label, record) -> {
                PerformanceRecord saved = performanceRecordRepository.save(record);
                i.addAndGet((int) saved.getN());
            });
            log.info("analysis report:{} has {} records, {} samples, used {}ms", reportId, sampleStates.size(), i.get(), System.currentTimeMillis() - start);
            sampleMap.get(reportId).clear();
        });
    }
}

3.4.3 Report Presentation

Aggregated data is visualized in real‑time dashboards, API endpoints, and performance curves.

Future Outlook

The platform initially operated independently of other company services, but has gradually integrated unified authentication, Feishu messaging, and distributed caching. It now supports core business performance testing with over 2,000 test reports per month and will continue evolving with AI and file distribution improvements.

4.1 File Distribution Upgrade

Current file distribution relies on MongoDB GridFS, which becomes a bottleneck at large scale; the plan is to adopt a hybrid MongoDB + OSS approach to handle >500 agents efficiently.

4.2 Further AI Exploration

AI techniques are being explored for performance model accuracy and proactive issue detection, with future work targeting AIGC‑driven problem localization and automated report generation.