Zero‑Code Scripted Guidance for Mobile Apps Using CV and AI

Background – Rapid app updates in the Internet industry require efficient user‑mindset building. Low‑code/zero‑code concepts are gaining traction as a way to accelerate digital transformation. Meituan’s delivery platform faces a growing number of business lines and frequent feature releases, creating a need for scalable, cost‑effective guidance solutions.

Current Situation – Traditional advertising, slogans, and offline promotion are costly and imprecise. In‑app functional guidance offers low cost, high coverage, and reusability, but existing implementations are labor‑intensive and technology‑stack dependent.

Goal & Challenges – Provide a solution that enables product and operation teams to create and launch guidance scripts without code, independent of the underlying tech stack (Flutter, React Native, Mini‑Program, PWA, etc.). Challenges include stack‑agnostic execution, high success rate, and zero‑code authoring.

Overall Design – The ASG (Application Scripted Guidance) system consists of three parts: terminal side, management backend, and cloud services. The terminal side handles pre‑processing, real‑time image matching, task scheduling, and multimedia rendering. The backend offers script editing, protocol configuration, preview, and permission control. Cloud services host resources and drive dynamic script distribution.

Technical Solution – Visual‑Intelligent Region Localization – To locate UI elements across heterogeneous stacks, the system captures a full‑screen screenshot and applies image‑matching algorithms. Traditional CV methods (ORB, SIFT, SURF) are combined with deep‑learning models to improve robustness. Feature‑point density, scale/rotation invariance, and memory constraints are addressed through multi‑scale pyramids, ORB‑based FAST detection, and rBRIEF descriptors.

Image Matching Workflow – The pipeline extracts features, matches them using locality‑preserving matching (LPM), and computes the target region’s coordinates. Experiments show that 5 000–10 000 feature points balance accuracy and performance on mobile devices.

Deep‑Learning Augmentation – For cases where traditional CV fails, a GlobalTrack‑style single‑stage detector is employed. It extracts the target region’s features, modulates the full‑screen feature map, and locates the region efficiently on‑device using the MTNN inference engine.

Robustness Guarantees – Execution success is monitored via image similarity thresholds, container‑route URL comparison, and timeout handling. Failure cases trigger retries or aborts. The system also filters or blocks interfering pop‑ups using platform‑specific hooks such as NavigatorObserver. didPush in Flutter and JavaScript injection for Web.

Zero‑Code Script Authoring – Recording SDK captures user actions, page metadata, and audio to generate key‑frames. A standardized protocol flattens the script into a hash map, enabling non‑technical users to edit, reorder, and publish scripts through a visual editor.

Results – Since November 2021, ASG has powered over 20 business scenarios, reducing script development effort to < 0.5 person‑day and improving task completion rates from 18 % to 35.7 %. Cost reduction exceeds 90 % while effectiveness gains reach ~20 %, yielding an overall efficiency multiplier of about 12×.

Conclusion & Outlook – The project demonstrates that a CV + AI hybrid approach can deliver stack‑agnostic, zero‑code guidance with high robustness and low cost. Future work will focus on richer script composition, dynamic timing, and deeper integration with rule engines.