Automated Testing Exploration and Practice in Meituan Waimai: Introducing the AlphaTest Platform

With the rapid growth of Meituan’s “to‑home” business, the system complexity and the number of test cases have increased dramatically. In the past two years the number of test cases has roughly doubled, exceeding 12,000, and developers now spend about half of their time on testing instead of pure development. This makes the introduction of automated testing essential.

Project Background : Meituan Waimai supports multiple business lines (food delivery, flash purchase, pharmacy, errands, etc.) and runs on diverse technology stacks such as native code, Mach (Meituan’s dynamic framework), React Native, Meituan mini‑programs and H5. The variety of stacks and the “three‑terminal reuse” (Waimai App, Meituan App, Dianping App) increase the difficulty of testing.

Project Goals : Build an automation solution with zero learning cost, low maintenance, high availability, and support for the multi‑business, multi‑App and multi‑technology‑stack scenario.

Solution Selection : Evaluated three representative tools – Appium, Airtest Project and SoloPi – and found that none satisfied the requirements for data recording, environment simulation, maintenance cost, and cross‑App reuse.

Practice and Exploration :

Challenges: Recording‑playback approaches are hard to edit, leading to high maintenance cost.

Pre‑condition preparation: One‑click environment simulation to set up accounts, mock data, location, etc., separated from the main test steps.

Data consistency: Both local cache and network data are cleared/recorded to ensure identical conditions between recording and playback.

Operation consistency: Multi‑modal element locating (ViewPath + image + coordinates) guarantees stable interaction across screen resolutions.

Traceable testing: Every instruction’s execution status and screenshots are recorded; failures generate detailed reports and videos for root‑cause analysis.

Case maintenance: Visual instruction editor allows adding, deleting or modifying steps without re‑recording; supports minor business changes and major UI refactors.

Cross‑App playback: SDK detects differences (pre‑condition, initial page, AB experiments, API mapping, scheme mapping) and automatically adapts the recorded case to another App.

Event‑point (埋点) testing: Records and validates event‑point timing and parameters, with configurable field‑rule checks.

Testing Process : The core flow consists of automated task triggering, replay cluster scheduling, assertion service, and message push. Cases are recorded on a device, uploaded as semantic instructions, and later replayed on a cluster of machines with retry mechanisms at task, node and sub‑task levels.

Automation Task Trigger : Integrated with the agile delivery platform to automatically launch test cases after a build.

Replay Cluster Scheduling : A 24‑hour machine pool splits a large replay job into sub‑tasks, each handling device allocation, app installation, authorization, scheme launch and result reporting, with configurable retry counts.

Assertion Service : Supports both text and image assertions. Image comparison uses a Siamese‑network based model (ResNeXt‑50 backbone, ContrastiveLoss) achieving >99 % accuracy. Failed assertions are categorized for automatic model retraining.

Message Push : Test reports are pushed in real time via Meituan’s internal message‑queue and OA SDK, with customizable templates to avoid notification fatigue.

Landing and Practice : AlphaTest has been deployed in more than 15 Waimai versions, covering 70 % of regression cases across native, Mach, React Native, mini‑program and H5 stacks. It supports UI automation, event‑point automation, dynamic‑load success‑rate testing and accessibility testing. The platform is continuously evolving toward smarter, more precise testing.