iOS Remote Control Solution: Architecture, Technology Research, and Implementation Details

Mobile remote control aims to allocate phone resources more efficiently and offer richer choices to users. The team has built an Android STF platform and, after extensive research and trial‑and‑error, completed basic iOS remote‑control functionality for user consumption.

The main goals for browser‑based iOS remote control are real‑time, smooth video streaming; real‑time interaction with the device; provision of common utilities such as live logs and app installation; and minimal hardware resource consumption.

Several iOS automation drivers were evaluated, including WedDriverAgent, Apple’s XCTest/XCUITest, Appium, and UIAutomation. UIAutomation was discarded because it was deprecated after Xcode 8. Appium relies on WDA (WebDriverAgent) which itself uses XCTest/XCUITest, so the team chose Facebook’s open‑source WebDriverAgent as the iOS driver solution.

Screen Capture Options

1. iOS‑Minicap – Provides smooth 30 fps video but cannot handle multiple devices on a single Mac because it uses AVFoundation and CoreMediaIO.

2. AirPlay Mirror – Apple’s proprietary, encrypted protocol; reverse‑engineering is difficult, so it was not pursued.

3. idevicescreenshot – Part of libimobiledevice; captures screenshots at 3‑5 fps, slower than needed, and would require custom modifications to improve throughput.

4. Custom WDA – Extending WDA to add screenshot and control capabilities via WebSocket offers a simpler maintenance path.

Architecture Design

The solution supplements an existing device‑management platform with an iOS remote‑control module. It consists of a master web service and an agent running on each device. The browser client communicates with the agent via WebSocket; libimobiledevice’s iproxy forwards data between the Mac host and the iOS device.

Detail Solutions

1. Screenshot Efficiency

By forking guadaran/WebDriverAgent , a faster screenshot method was found, achieving 20‑30 ms per frame. Because large images increase network I/O, compression is applied. Depending on device resolution, a scale factor (e.g., 0.4 for iPhone X) keeps capture time under 40 ms while preserving clarity.

2. WDA Resource Consumption

Continuous high‑frequency screenshots increase CPU and memory usage, potentially causing crashes. The team balances capture interval with device performance and releases image objects using @autoreleasepool {} to limit memory growth.

3. WebSocket Communication

WebSocket support was added to WDA with open/close interfaces. During remote control sessions, the connection can transmit compressed images, and duplicate frames are suppressed to reduce network traffic.

4. Operation Events

WDA’s default tap, swipe, and drag commands have synchronous waits (2‑5 s). By replacing them with the private API XCEventGenerator.h , latency drops to near‑instant execution. Many actions can also be simulated via pressAtPoint .

Future Plans

Achieve video streams above 20 fps and millisecond‑level event latency; a single Mac mini can handle 10+ iOS devices, providing a near‑real testing experience.

Continue improving iOS usability, proxy configuration, multi‑device coordination, and integration with UI automation testing frameworks.