5 Open-Source AI Projects That Automate Mobile Phones

Traditional mobile automation relied on tools like Appium or Airtest, which required developers to know low‑level UI element IDs (resource‑id, xpath). When an app updates and IDs change, scripts break. Recent large‑model AI, especially vision models, now make AI‑driven phone manipulation feasible.

MobiAgent is an open‑source framework from the IPADS lab designed for AI‑controlled mobile interaction. It can, for example, search for the best‑selling men’s jeans on Xiaohongshu, locate the same item on Taobao, and send the brand, name, and price via WeChat, or order a lemon tea on Ele.me. The system splits tasks into three specialized modules: Planner (creates the overall plan), Decider (chooses the next click), and Grounder (pinpoints the exact screen location). Its core components are the MobiMind model family, the AgentRR acceleration framework that dramatically reduces repeated‑execution latency, and the MobiFlow benchmark platform covering 10+ mainstream apps (Xiaohongshu, Gaode Map, Ele.me, Taobao, etc.) with simple and hard difficulty levels, providing standardized performance scenarios.

开源地址：https://github.com/IPADS-SAI/MobiAgent

Mobile‑Agent (Alibaba) can see the screen, click buttons, and operate across apps without relying on complex system back‑end interfaces; it mimics human visual interaction. Example: "Search Jinan travel guides on Xiaohongshu, sort by collection count, and save the first note." The framework captures screenshots via ADB, feeds them to a multimodal model, which decides the next click, swipe, or text input.

开源地址：https://github.com/X-PLUG/MobileAgent

Droidrun is an AI‑based Android/iOS automation Agent framework with 6.2K GitHub stars. Its core philosophy is to let AI handle the reasoning while the framework executes actions, breaking the traditional dependence on specific UI controls. Example task: "Search for the cheapest two‑person apartment in San Francisco next week and return the cheapest option."

开源地址：https://github.com/droidrun/droidrun

AppAgent (Tencent) is officially titled "Multimodal Agents as Smartphone Users". It aims to give AI agents human‑like perception and operation abilities. The system captures screenshots via ADB, sends them to a multimodal large model, which analyzes UI elements and decides whether to click a coordinate or swipe, achieving true visual interaction. Unlike other projects, AppAgent mimics how humans learn new software: it can explore autonomously or watch human demonstrations, recording each button’s function into a Knowledge Base. When executing a task, it consults this documentation, allowing precise actions without re‑learning each time.

开源地址：https://github.com/TencentQQGYLab/AppAgent

mobile‑use (Minitap AI) has 1.8K stars and supports both Android and iOS. Its core logic lets an AI model simulate human actions: it captures the current screen, sends the screenshot and the user’s natural‑language command to a multimodal model, which returns concrete coordinates or commands (click, swipe, input). mobile‑use translates these into ADB commands for execution, then re‑captures the screen to verify progress until the task completes. The project integrates the Maestro mobile testing framework as the underlying interaction engine for stability and allows various model back‑ends (OpenAI API, locally deployed models, etc.).

开源地址：https://github.com/minitap-ai/mobile-use