Phone-Harness: Let AI Agents Control Your Phone — Tap, Read, No Jailbreak Needed

What Phone-Harness Is

Phone-Harness is a Python command-line tool plus a SKILL.md instruction manual that gives any command-running agent (Claude Code, Codex, etc.) the ability to tap, swipe, type, and read text on a real phone. The phone side requires no app install, no jailbreak (iOS), and no root (Android).

Four Connection Paths to the Phone

Personal iPhone : Uses macOS 15's built-in iPhone Mirroring window; screenshots are OCR'd via Apple's Vision framework, and click/input events are injected into that window.

Desktop Android : Uses ADB over USB or same-network Wi-Fi; text comes from the system accessibility tree.

Cloud Android : Official hosted emulators, also accessed via ADB.

Cloud iPhone : Official hosted real iPhones plugged into Macs; commands sent over HTTPS.

All four paths expose the exact same agent-facing interface. A script written for a cloud phone runs on a local device by changing only config set platform.

Installation in Three Commands

The tool is designed so an agent can install itself. A single prompt suffices:

Install or upgrade phone-harness to the latest stable version with uv using Python 3.12, register the skill from phone-harness skill, then read onboarding.md and walk me through it.

Manual install:

uv tool install --python 3.12 --upgrade --force phone-harness
# or pip install phone-harness
mkdir -p ~/.claude/skills/phone-harness
phone-harness skill > ~/.claude/skills/phone-harness/SKILL.md
phone-harness doctor

Requires Python 3.10+. The iPhone path additionally needs macOS 15 and two system permissions: Accessibility (for clicks/typing) and Screen Recording (for screenshots); Screen Recording changes require restarting the Python process.

The Agent Toolset: 20+ Concrete Functions

The SKILL.md manual lists over 20 specific functions, not vague descriptions. Key examples:

Observe : screen() returns all visible text with coordinates; find_text("Weather") locates specific text; screenshot() captures a PNG; wait for target text to appear (with timeout); current_app() identifies foreground app.

Act : tap_text("Weather") taps by text label; tap(x, y) taps raw coordinates; type_text("hello") types text; swipe("up") swipes once; open_app("Notes") launches an app. screen() is the core: it returns a list of objects {text, confidence, source, x, y, w, h} — each visible text element with its center coordinates. Agents filter this list in Python, which is far more reliable than guessing coordinates from a screenshot. On Android the boxes come from the accessibility tree (exact); on iPhone Mirroring they come from Apple Vision OCR.

List scrolling is encapsulated: scroll_until(condition) scrolls until a predicate passes; scroll_collect() scrolls to the bottom, deduplicates rows, and returns items, stop reason, and scroll count.

A complete heredoc example:

phone-harness << 'PY'
open_app("Notes")
tap_text("New Note")
type_text("hello from the harness")
print([o["text"] for o in screen()][:10])
PY

Helper functions are preloaded from helpers.py; agent-written helpers persist in ~/.phone-harness/agent-workspace/agent_helpers.py and survive CLI upgrades.

The Engineering Inside the Manual

The 35 KB SKILL.md solves not "how to tap" but "how the agent knows the tap worked." Phones have no DOM, no return values; most failures are silent no-ops — the tap does nothing and the agent is unaware. The manual enforces hard rules:

State expected change before acting. Before a tap, articulate what should appear — a title, a row, text in an input. If you can't write it, you can't judge success.

Do one thing, verify one thing. After acting, wait for the target text or app to appear, then inspect the text list. No sleep for page loads.

Validate sequences before batching. Batch a sub-task into one script for speed, but end with a cheap check.

Isolate failures; don't re-run the whole batch. Re-running hoping for luck is the most common agent error.

A subtle navigation semantic is spelled out: scroll describes the content direction you want to see; swipe describes the finger direction . To see content below, use scroll("down"); to advance a video feed, use swipe("up") — both map to the same physical gesture. All four directional functions use content direction except swipe.

Safety boundaries are hard-coded: before sending messages, posting, placing orders, paying, deleting, or changing settings, the agent must pause and ask the human. It never enters PINs, passwords, or 2FA codes. On cloud phones, cloud open returns control to the human for credential entry.

Cloud Phone Service Model

The open-source harness is free; the company sells a managed cloud phone service on top.

Cloud Android : 5¢/minute, per-minute billing. New accounts get $5 credit (~100 minutes). Two types: ephemeral (fresh each session, destroyed on exit) and persistent (launched with a profile_id; apps, logins, files, settings preserved; ~10-second resume).

Cloud iPhone : Real devices hosted on Macs. Not per-minute; data persists until explicit reset. Requires separate application.

For agent-as-a-service architectures: split into an orchestrator (holds API key, creates sessions, hands host:port and unlock code to a sandbox ) and the sandbox (runs adb connect with no access to other devices, no billing rights, cannot end sessions). If the agent misbehaves, the orchestrator rotates the unlock code or calls DELETE /sessions/{sid}.

Cloud path also supports MCP: add https://mcp.phone-harness.com/mcp to any MCP client (currently only for cloud Android and cloud iPhone).

Costs and Ceilings

OCR reads text, not icons. Buttons without text labels require a vision-capable model, limiting full autonomy — many critical app actions are icon-only.

No multi-touch, no camera, no Face ID flow. DRM video renders black; streaming tests see a black screen.

Lock screen requires a human. iPhone unlock breaks mirroring; locked Android needs a password. Connecting the phone remains the user's job; the tool only reports where doctor gets stuck.

iPhone path locked to macOS 15+. No Mac? Use Android or cloud iPhone.

Cloud Android is an emulator, not a real device. Scenarios needing real device fingerprints, real carrier numbers, or carrier-specific behavior won't work. Cloud iPhone is real hardware but requires application and per-device provisioning.

Who Should Use It (and Who Shouldn't)

Best fit for two groups:

Mobile app developers. Let an agent walk registration, checkout, settings — capturing screenshots and screen text at each step — for regression without manual tapping. Appium and uiautomator2 still exist, but Harness lets agents describe tasks in natural language; no pre-written scripts.

Anyone verifying "what it looks like on a real phone." Open a web page or email template on a real device, screenshot, read text — more accurate than browser device emulation.

Poor fit for: fully unattended high-volume batch jobs (OCR and lock screen are hard walls); and non-Mac users who only have a personal iPhone.

A guiding principle from the manual's opening, which the author calls more practical than any feature:

If you can do it on a computer or the web, don't touch the phone. Use the API if there is one; use the web version if it exists. Reserve the phone for things that truly only exist on the phone — app-only services, flows tied to phone numbers and SMS codes, and cases where you must see the real-device rendering.

That statement sounds anti-commercial, but it's the prerequisite for this class of tool to remain useful long-term.

Project repository:

https://github.com/ShawnPana/phone-harness