How Huolala Created an Intelligent Automated Testing System to Raise Coverage & Cut Regression Costs

1. Background and Challenges

As the market changed and business expanded, Huolala grew from freight services to ride‑hailing and international domains. New services are built on existing system architecture to accelerate incubation, but this raises higher standards for quality assurance.

Redundant code: Deprecated business logic or new services modified from old systems generate large amounts of redundant code, making it difficult to allocate manpower for cleanup and complicating test coverage monitoring.

High regression cost: Rapid releases demand high‑coverage automated tests to replace most manual regression, allowing more resources to focus on testing new code.

Lack of automation metrics: There is no quantitative way to assess whether interface automation can replace full regression, leading to inconsistent standards and quality risks across services.

2. Overall Solution

To address these issues, we aim to build an intelligent automated testing system that analyzes effective code, performs code‑link analysis for each service and interface, offers smart test case supplementation recommendations, and standardizes semi‑automated test script generation. The system also provides intelligent monitoring of automation progress, coverage change statistics, and defect management.

Automation construction guidance

Automation is a large, long‑term effort; prioritizing tasks yields higher returns.

Given many services contain redundant code, we need the ability to exclude redundant code when calculating effective coverage, then use an algorithm to guide automation construction.

Test case authoring empowerment

To reduce the cost of writing interface automation and achieve standardized test cases, we wrapped many helper functions into the existing framework, such as:

Authoring phase: Using the JsonUtils class to quickly generate request bodies, PICT class for pairwise parameter generation, MockUtils for mock parameter construction, etc.

Assertion phase: Various assertion utilities are packaged to quickly and consistently generate assertion code, ensuring no middleware or field is missed.

Construction progress visualization

Early on, without effective coverage monitoring, a service with massive redundant code showed a total coverage report of 30%+, but QA subjectively believed the automation was complete, making it impossible to identify gaps.

Monitoring is a critical process that provides valuable data for better decision‑making.

Key monitoring items include:

Service integration summary : Check which services have automation; core services must have automated tests.

Weekly effective coverage progress : Monitor automation coverage and aim for 80‑90% on core services.

Test case stability : Assess test quality; low stability may affect test accuracy.

Automation defects : Identify software quality issues; frequent defects indicate deeper testing needs.

Automation milestones : Define high‑priority services for test case supplementation and track milestone achievement.

3. Capability Building

Below we detail the implementation ideas of core features.

3.1 Redundant Code Coverage Solution

Can we calculate effective code coverage by removing redundant code?

Jacoco exclude package: Manual identification of unused code is labor‑intensive and error‑prone; maintaining exclusion lists is difficult.

Method‑level exclusion: Some methods in a class are still used while others are deprecated, but Jacoco’s official exclusion does not support method‑level filtering.

Counting ineffective code is a huge workload; can we instead count effective code?

When judging whether a service function is deprecated, we check if the interface has request records in recent months. Conversely, online monitoring can deem a business with traffic as effective. Combining service code‑link relationships, we filter the total effective code set and generate a coverage report, solving the heavy manual exclusion problem.

Additionally, manual configuration can add or exclude specific link code to handle edge cases such as traffic‑less effective interfaces.

To realize the effective code coverage detection, we need four capabilities:

Online application monitoring: Identify which interfaces or tasks have traffic and require testing.

Code‑link analysis: Determine the relationship between called interfaces/tasks and entry methods, including call‑graph relationships.

Generate coverage report for specified code: Replace the full‑code coverage report with the “effective code set”.

Intelligent coverage recommendation: Use algorithms to suggest test case supplementation, automatically constructing environment, request, and assertion code.

3.1.1 Effective Business Monitoring

Companies like SkyWalking collect Java HTTP and SOA call information and Kafka consumption details. Huolala already has an intelligent monitoring platform (Monitor) covering HTTP, SOA, Kafka, etc., with an Open API to fetch effective business data. By calling this API, we can gather interfaces with traffic, jobs with run records, and topics with consumption records within a time window, thus consolidating effective business for each service.

See Huolala technical article “Core Application Coverage 100%, Huolala Intelligent Monitoring Practice”.

Figure: Huolala Monitor capabilities

3.1.2 Code‑Link Parsing

The typical solution uses Byte Code Engineering Library (BCEL) to analyze Java bytecode and generate method call graphs.

BCEL reads class files, extracts class, method, and instruction information, identifies method call instructions, and builds a call graph by adding edges from the current method to the target method. This produces a complete call graph showing all method relationships.

BCEL is the key component for implementing code‑link functionality, enabling accurate analysis of Java bytecode and aiding developers in understanding program structure and behavior.

Huolala’s Precise Testing platform has realized code‑link parsing, extending it to support various call relationships and generate call graphs. Further details will be introduced in future Precise Testing practice.

3.1.3 Effective Code Coverage Report

Jacoco’s official version cannot generate reports for specific code sets, but a customized Jacoco can achieve this.

Core modifications:

Pass the effective code set to the CoverageBuilder when creating the report.

@Option(name = "--effecCode", usage = "effective code for report", metaVar = "<file>")
String effecCode;

private IBundleCoverage effecAnalyze(final ExecutionDataStore data, final PrintWriter out) throws IOException {
    CoverageBuilder builder;
    builder = new CoverageBuilder(this.effecCode);
    final Analyzer analyzer = new Analyzer(data, builder);
    for (final File f : classfiles) {
        analyzer.analyzeAll(f);
    }
    printNoMatchWarning(builder.getNoMatchClasses(), out);
    return builder.getBundle(name);
}

Modify ClassProbesAdapter.visitMethod to determine if the code belongs to the effective set.

@Override
public final MethodVisitor visitMethod(final int access, final String name,
      final String desc, final String signature, final String[] exceptions) {

   final MethodProbesVisitor methodProbes;
   final MethodProbesVisitor mv = cv.visitMethod(access, name, desc,
         signature, exceptions);
   // Effective code coverage
   if (mv != null && isContainsMethod(name, desc, CoverageBuilder.classInfos)) {
      methodProbes = mv;
   } else {
      methodProbes = EMPTY_METHOD_PROBES_VISITOR;
   }
   return new MethodSanitizer(null, access, name, desc, signature, exceptions) {

      @Override
      public void visitEnd() {
         super.visitEnd();
         LabelFlowAnalyzer.markLabels(this);
         final MethodProbesAdapter probesAdapter = new MethodProbesAdapter(
               methodProbes, ClassProbesAdapter.this);
         if (trackFrames) {
            final AnalyzerAdapter analyzer = new AnalyzerAdapter(
                  ClassProbesAdapter.this.name, access, name, desc,
                  probesAdapter);
            probesAdapter.setAnalyzer(analyzer);
            methodProbes.accept(this, analyzer);
         } else {
            methodProbes.accept(this, probesAdapter);
         }
      }
   };
}

With these steps, the system determines whether code is effective and generates a coverage report accordingly.

Similarly, for a single‑interface coverage report, provide the interface name and pass the related code set to the report.

3.2 Intelligent Test Case Supplement Recommendation

3.2.1 Sorting Uncovered Code Lines

Assume a service has two uncovered methods: Method A with 50 lines and Method B with 5 lines. Writing a test case for Method A covers more lines, so its priority is higher, ignoring other factors.

In practice, we still need to consider service importance, interface importance, etc., to decide which uncovered code should be prioritized for test case supplementation.

Manually scanning a Jacoco report to find which logical branches have many uncovered lines is impractical.

We parse the Jacoco report.xml to obtain uncovered line counts per method, then sort methods by uncovered lines descending, and push the highest‑line methods to QA for priority test case creation.

Example XML snippet:

"method":{
        "counter":[
            {"@type":"INSTRUCTION","@missed":"129","@covered":"244"},
            {"@type":"BRANCH","@missed":"15","@covered":"15"},
            {"@type":"LINE","@missed":"33","@covered":"62"},
            {"@type":"COMPLEXITY","@missed":"11","@covered":"5"},
            {"@type":"METHOD","@missed":"0","@covered":"1"}
        ],
        "@name":"getMemberNewList"
    }

3.2.2 Recommendation Algorithm

Writing comprehensive, effective automated test cases is challenging. To address this, we propose a multi‑factor intelligent test case supplementation algorithm.

The algorithm considers service level, interface level, interface traffic, and the number of code lines each test case can cover. Service and interface levels reflect importance; traffic reflects usage frequency; line coverage reflects test breadth. Additional factors such as code complexity, historical defect data, and change frequency can further improve recommendation accuracy.

Each uncovered code segment receives a score based on these factors; test cases are then ranked by score, and the highest‑scoring cases are recommended to QA, enabling more precise and efficient test case supplementation.

3.3 Test Case Authoring Assistance Tools

Interface automation test case authoring is divided into five stages: initialize test environment → build request parameters → send request → verify results → clean up data. Our assistance tools provide efficiency gains at each stage. Example: unban driver test case.

@Test(description = "Unban driver demo")
public void testUnban01() throws Exception {
    // Initialize test environment
    RandomDriverConstants RandomDriverConstants = new RandomDriverConstants();
    randomDriverService.initData(RandomDriverConstants);
    driverAdapter.updateDriverStatus(RandomDriverConstants.driverId, "BANNED_STATUS");

    // Build request parameters
    String req = jsonUtils.fileToStr("UnbanReq.json");
    req = jsonUtils.reqChangeKeyOfValue("driverId", RandomDriverConstants.driverId, req);

    // Send request
    Result result = requestUtil.transportPost("unban", req, Result.class);

    // Verify results
    assertion.verifyEquals(result.getRet(), "0");
    databaseQueryService.check("driver_status_table", "UNBAN_STATUS");
    redisQueryService.check("driver_cache", "UNBAN_CACHE");
    kafkaReceiver.check("driver_status_topic", "UNBAN_MESSAGE");
    loggerQueryService.check("unban_log", "UNBAN_LOG");

    // Clean up test data
    randomDriverService.clear(RandomDriverConstants);
}

Other helper utilities include:

CreatePICT: Pairwise combination testing using Microsoft’s PICT tool.

// Generate full combination of account_type and packet_type
@DataProvider
public static Object[][] get_coupon_packet_list_packet_type() {
    String content = "account_type:4,3
" +
            "packet_type:1,2,3,4,6,8
" +
            "IF [account_type] = 3 THEN [packet_type] = 3 ;";
    Object[][] ob = IdUtil.getPictOnQamp(content, "2");
    return ob;
}

MockUtils: Mock platform based on JVM‑Sandbox for injecting exception scenarios.

// Mount member mock
mockUtil.onlineMemberMock();
Integer id = mockUtil.addMock(appId, mockClass, mockMethod, returnObj, ruleConfig);
mockUtil.updateMock(id, appId, mockClass, mockMethod, returnObj, ruleConfig);
SOAResult soaResult = requestUtil.memberCenterSOAPost(path, method, req, SOAResult.class);

3.4 Monitoring Dashboard Construction

Based on the capabilities and quality‑management data, we transform service integration, coverage progress, and defect metrics into front‑end charts.

4. Practice Results

4.1 Automation Process Improvements

System intelligent test case recommendation

Before: QA manually selected important services, interfaces, and scenarios based on business familiarity.

Now: The system automatically recommends test cases to QA based on effective coverage and other standards, greatly improving selection accuracy and efficiency.

Inherited framework assistance

Before: Test case environment setup, parameter construction, result assertion, and data cleanup required lengthy code.

Now: Leveraging the inherited framework, test cases are written concisely, boosting development speed and maintainability.

Automation coverage monitoring

Before: QA self‑reported automation levels without systematic progress monitoring.

Now: The platform periodically aggregates automation coverage and improvement percentages, enabling governance of automation progress and resource allocation.

4.2 Automation Benefits

Effective coverage increase: Over 40 services improved their effective coverage by more than 20% within a month.

Automation defect ratio: The proportion of defects found by automation to total defects reaches around 17%.

Automation labor savings: Services with existing automation save about 60% of full‑regression manpower per iteration.

One typical transaction service increased effective coverage from 45% to 91% in a month after continuously adding platform‑recommended test cases; remaining uncovered lines are only exception handling code, indicating automation can replace full regression.

Figure: One service’s monthly automation coverage improvement

5. Future Outlook

Future directions for intelligent testing include:

Effective coverage statistics applications: Apply effective coverage metrics beyond automation, such as manual testing, traffic replay, precise testing, or other quality‑efficiency projects.

Coverage‑traffic isolation: Use traffic tagging to separate coverage generated by manual testing, automation, traffic replay, etc., ensuring they do not interfere.

Intelligent test case generation: Explore AI models, data‑driven, NLP, reinforcement learning, and code‑analysis strategies for generating test cases, further enhancing intelligent testing efficiency.