Control Your iPhone via USB with Natural Language: iPhone Use Turns Codex into a Mobile Automation Agent

Overview

iPhone Use (by zhongerxin, MIT license, Python) lets you automate a real iPhone by connecting it via USB to a Mac and issuing natural-language commands to Codex. It replaces fragile Appium scripts with a local MCP service that bridges Codex to WebDriverAgent running on the device. No jailbreak or separate Appium server is required.

Architecture & Toolset

The system runs a local MCP service ( iphone_use) on macOS. It forwards USB traffic to WebDriverAgent (fixed at version 16.14.0) on the iPhone and exposes 17 tools (prefixed pua_ for Phone Use Agent) to Codex, plus two tools for the live screen sidebar. Tools are grouped as:

Environment & startup: pua_doctor (diagnostics), pua_setup (device config, background install/launch), pua_ready (readiness check).

Page inspection: pua_observe (element tree / screenshot / both), pua_find (locate target).

App management: pua_apps (bundle IDs, install evidence), pua_launch_app (launch).

Interaction: pua_tap (click), pua_swipe (swipe), pua_press_button (Home, etc.).

Input & wait: pua_type_text (Unicode input, auto-chunked with resume), pua_wait (bounded wait).

Composed actions: pua_batch (execute known step sequence in one call), pua_scroll_find (scroll once to find clickable target, then pause with screenshot), pua_collect_list (bounded scroll collection with deduplication and coverage report).

Preview & stats: pua_screen (preview toggle), pua_metrics (timing stats).

Key Tools in Practice

pua_collect_list

scrolls once per screen, reads, deduplicates, and stops at the boundary, returning what has been covered and what remains unconfirmed — more reliable than letting the model scroll and track manually. pua_batch bundles a fixed sequence of taps, inputs, and waits into a single call, cutting model round-trips for deterministic flows. pua_scroll_find slides at most once; if no clickable target appears, it returns a screenshot for human review instead of blind scrolling.

Fallback: Screenshot-Based Coordinate Tapping

When element-tree location fails (element missing, obscured, or unclickable), the tool returns a screenshot plus a conversion factor image.pixel_to_point. The model multiplies pixel coordinates by this factor to get iOS point coordinates, then calls pua_tap with those points. Rules prevent blind retries: scroll-find slides once then pauses; occlusion, scroll stall, input mismatch, or missing expected page all trigger screenshot fallback; existing screenshots are reused rather than re-captured.

The project explicitly avoids re-playing clicks, inputs, or submits after timeouts or disconnects — it reads actual state first.

Installation Requirements

macOS + Codex desktop or CLI

Full Xcode (not just Command Line Tools) with first-launch configuration done

Apple account and development team usable in Xcode

Real iPhone connected via USB (trusted, developer mode enabled if needed, unlocked during install)

Python 3.9+ (standard library only)

Node.js 20.19+ / 22.12+ / 24+, npm 10+

Xcode must support the iPhone's iOS version. Installation is two-step: (1) paste a prompt into Codex to clone the repo and run sh scripts/install.sh, which registers the MCP service and installs the plugin/skills; (2) reconnect Codex and run the iphone-use-setup skill to detect device, sign with your team, build and launch WebDriverAgent until pua_ready returns ready=true. First-time device steps (Apple ID login, trust, developer mode, unlock) require manual interaction. Subsequent runs reuse configs and builds from ~/.local/share/iphone-use/ (permissions 700/600). Update via git pull --ff-only && sh scripts/install.sh then reload Codex chat.

Design Decisions That Reduce Model Overhead

Version 0.2.0 introduced three optimizations reflecting the insight that every extra model turn costs latency and error risk:

Compact observation output: Strips XCUIElementType prefixes, omits name / value when identical to label, compresses rect to integer array [x,y,width,height]. A 99-node page dropped from 18,424 to 6,868 bytes (~60% reduction).

Optimistic execution: Assumes a normal action response means success; does not call observe after every tap, navigation, or Home press. The next observation call doubles as verification.

Guaranteed screenshot delivery: Earlier Codex versions dropped image blocks when structured content was present. The fix returns a single compact JSON text block plus one image block, ensuring the model actually receives the fallback screenshot.

Suitability & Telemetry

Best for developers who need repeatable iPhone automation without Appium scripting, and for those integrating phone control into Codex-style agent workflows. Core advantage: fully local USB channel — data, signing, builds stay on machine.

Not suitable for: non-macOS users, Android users, one-off tasks (manual faster). Learning curve is high due to Xcode provisioning and real-device signing prerequisites.

Anonymous usage stats (launch, tool calls, connection state, latency, error categories) are sent with a random install ID; no screen content, input text, or device IDs. Disable with IPHONE_USE_ANALYTICS=0 or DO_NOT_TRACK=1 and restart the MCP service.

Project: https://github.com/zhongerxin/iPhone-use (2.4k+ stars, updated 2026-10-10).