Automating AB Test Validation for Mobile Apps Using UI Test Framework and Enterprise WeChat Bot

Author: Zou Delong

Idea Birth

Based on the internal UITest framework integrated with the Zhaoliangji app, over 70 UI scenarios and 140 data‑point verification cases have been accumulated and are already in production. Although the existing AB‑test workflow has improved manual verification quality and speed, it still requires 8‑10 hours of repetitive human effort for each experiment (three groups across Android and iOS). The article explores how UI automation can increase efficiency and solve this problem.

Action Plan

1. Collaborate with the big‑data team to confirm the standard AB‑test acceptance process and identify which steps can be automated:

Mobile device proxy

Local host proxy

Configure API to refresh experiment cache (feasible)

Home‑page AB‑group interface verification (feasible)

AB scenario reporting interface (feasible)

Conversion path execution (feasible)

Re‑validation after AB switch (feasible)

Order/payment script execution (feasible)

The current client‑side AB flow is illustrated in the following diagram:

2. Design the solution: after confirming data and feasibility with the big‑data team, extend the existing UITest framework by adding Python + request scripts and a configuration‑file approach for business users, and integrate an Enterprise WeChat robot to report the entire acceptance process.

Integrate real‑time reporting via Enterprise WeChat bot

Add SQL output for manual review

Process Details

All implementations are built on the existing UITest framework, though some manual steps remain.

(1) Data preparation – manual configuration is still required, but creating experiments and deploying AB services in the test environment are repeatable steps.

(2) Fill the configuration file with group IDs, experiment IDs, device IDs, and WeChat bot information.

# True: execute this group  False: skip this group
A = True
B = True
C = False
D = False
ab_id = 10xxx   # AB experiment ID
deviceId = 'xxxx-123-321--xxxxxxxxx'   # Device ID
# Debug bot
webHookTest = 'xxx'
webHookTest_key = 'xxx'

(3) Execute the script locally with Python.

# -*-coding:utf-8-*-
import os
import UiTest
from ZljUItest.ab_case import ab_regression_test

def test():
    ab_regression_test.ab_test_b2c()

if __name__ == '__main__':
    receivers = "xxxx"  # notification email
    root_path = os.path.abspath(os.path.dirname(__file__) + os.path.sep + "..") + os.path.sep
    config_path = root_path + "config.yaml"
    UiTest.run_ui(email=receivers, suit_file=__file__, project=root_path, os="android", config_yaml=config_path)

AB experiment automation sends notifications via Enterprise WeChat:

4. Acceptance execution flow – verify that the config contains the experiment ID to ensure the AB configuration is effective.

def ab_config(self, ab):
    """
    After enabling the experiment, refresh its cache data (must refresh after each change).
    :param ab: experiment ID
    :return:
    """
    url = "xxx"  # API address
    response = requests.get(url)
    code = response.status_code
    text = response.text
    ab_text = re.findall(r'"test_id":{}.*?0,'.format(str(ab)), text)  # extract data
    Logger().setlog(str(code))
    Logger().setlog(str(ab_text))
    ab_text = str(ab_text)
    if str(ab) in ab_text:
        ab_text = ab_text.replace("['", "{").replace(",']", "}")
        ab_text = json.loads(ab_text)
        Logger().setlog("Print AB info from config file")
        Logger().setlog(str(ab_text))
        Logger().setlog("ab_config succeeded for experiment ID:{}".format(str(ab)))
        ABPage().webhook_push(host=zljconfig.webHookTest, text="ab_config succeeded for experiment ID:{}, starting APP AB check...".format(str(ab)))
        return True
    else:
        Logger().setlog("ab_config failed for experiment ID:{}".format(str(ab)))
        ABPage().webhook_push(host=zljconfig.webHookTest, text="Experiment {} not found".format(str(ab)))
        return False

Switch AB groups, automatically cover all group validations, restart the app (cold start), and verify that the data points are correctly reported and sequenced.

if zljconfig.A:
    ABPage().webhook_push(host=zljconfig.webHookTest, text="Experiment has A bucket, starting A bucket validation")
    # Switch device to A bucket
    ABPage().switch_group_id(ab_id=ab_id, device_id=zljconfig.deviseId, hit_num='A')
    device.stop_app(package="xxxxx")
    log_path = ABPage().ab_get(ab_id=ab_id, clear=False)
    device.start_app(package="xxxxx")
    time.sleep(10)
    assert ABPage().ab_assert(log_path, ab_id=ab_id, hit_num='A')
    # TransitionPath().b2c_one_path()

After script execution, results and review SQL queries are pushed via the Enterprise WeChat bot:

Future Improvements

Add failure notifications to partners when assertions fail.

Use configuration files to assemble conversion paths and data‑point pages, covering more business lines.

Integrate with Jenkins for visual configuration editing and script execution, enabling business users to adopt the solution quickly.

Results

Data analysis shows the first version improves efficiency by over 50 %, though currently limited to home‑page AB verification. Ongoing work will add more scenarios and support iOS clients.

Conclusion

“To do a good job, one must first sharpen the tools.” UI automation is a crucial part of mobile app development; after implementation, consider broader business scenarios and apply it to testing workflows to boost efficiency and empower teammates beyond core processes.