How pi-computer-use Lets AI Control macOS and Windows Apps Like a Human

What pi-computer-use Does

pi-computer-use is an extension for the Pi AI coding assistant that adds visual perception and interaction capabilities, allowing Pi to see the screen, click buttons, type text, and drag items on macOS 14+ and Windows via the system Accessibility APIs.

Core Functions

Window Discovery and UI Inspection

The find_roots command lists all open applications and windows. observe_ui takes a snapshot of the active window, exposing buttons and text fields. search_ui, expand_ui and inspect_ui let Pi locate elements by keyword, zoom into details, and verify clickability, returning a reference ID for later actions.

Action Execution

The act_ui command executes a sequence of steps such as “click the login button then type 123456”. Supporting commands include read_text, wait_for, keypress (which first tries semantic Accessibility actions before falling back to raw key events), scroll, drag, and set_text which replaces field contents without a prior select.

Fallback to Screenshots

When the Accessibility data is incomplete—for custom UI elements that hide information—pi‑computer‑use captures a screenshot and lets Pi locate targets visually, ensuring commands are not lost.

Browser Acceleration

For Chromium‑based browsers the extension can use the Chrome DevTools Protocol (CDP). The fast path AX-first focuses the address bar (Cmd+L), types a URL, and presses Enter, avoiding mouse movements and providing a stealthier, faster navigation.

Batch Actions and Stealth Mode

Multiple steps can be packed into a single act_ui call, reducing round‑trips. Enabling PI_COMPUTER_USE_STEALTH=1 forces the agent to stay on pure Accessibility paths, never taking foreground focus or mouse control, which allows background automation while the user works in another application.

Configuration and Quick Start

Install the Pi agent and the extension:

npm install -g @earendil-works/pi-coding-agent

pi install git:github.com/injaneity/[email protected]

On macOS grant Accessibility and Screen Recording permissions to the helper located at ~/.pi/agent/helpers/pi-computer-use/bridge. On Windows no extra helper is required.

After installation, run /computer-use inside Pi to view the active configuration, then try a simple query such as “show me the current windows and clickable buttons”. Pi will invoke find_roots and observe_ui and return the results.

Author’s Assessment

The author finds the extension reliable for tasks that lack public APIs, especially when combined with stealth mode, which lets Pi fill forms in Chrome while the user codes in VS Code without interference. The main limitation is the dependence on the target application’s Accessibility support; otherwise the screenshot fallback works.

Overall, pi‑computer‑use expands Pi from a terminal‑only code assistant to a desktop‑operating proxy, filling a niche for legacy software, browser pages, and occasional UI clicks.

GitHub: https://github.com/injaneity/pi-computer-use