HybridCUA: Teaching AI Agents When to Click and When to Command

Computer-use agents have traditionally relied on GUI-only interaction — perceiving screens, locating coordinates, and executing clicks, scrolls, and keystrokes step by step. While universal, this approach is slow and error-prone; a single misclick can cascade into task failure. Meanwhile, coding agents like Claude Code and Codex have demonstrated the power of CLI: a single shell command can replace dozens of GUI actions, but CLI lacks visual perception for tasks requiring interface feedback.

Real computer work demands both. An agent must decide when to click the interface and when to run a command, then verify the command's outcome. However, simply granting existing models terminal access backfires: on OSWorld, four representative agents all saw accuracy drop by 2.5–11.5 percentage points when CLI was enabled. The problem is not whether they can use a shell, but knowing when and how to use it.

HybridCUA Framework

HybridCUA addresses this with three innovations:

1. Unified Action Format

All executable actions are expressed as a single bash call. CLI commands are written directly in bash; GUI operations are embedded as pyautogui Python snippets inside the same bash invocation. At each step the model sees the latest screenshot and the previous command's output, allowing seamless chaining — e.g., close a menu via GUI, then read a file via CLI to verify.

A controlled experiment shows this unified format outperforms a separated design (distinct tools for GUI and CLI) by 7.2 percentage points on the same data.

2. Scalable Mixed-Trajectory Construction

The team built a reproducible data pipeline covering three execution modes:

Pure GUI trajectories from the open-source UI-MOPD dataset, with every UI action translated into equivalent pyautogui code.

Pure CLI trajectories generated by Qwen3.8-27B in a Claude Code environment. The team first compiled command-line skill manuals from software docs and reference code, then had the model solve CUA-Gym tasks using only the terminal, keeping successful trajectories.

Mixed trajectories from two sources: (a) letting the model freely switch between interfaces during execution; (b) collapsing "open terminal, type command character by character" segments in GUI trajectories into single full commands, re-playing them, and retaining only successful replays.

The final SFT corpus contains 5,023 trajectories across 11 application domains, averaging 10.4 steps each. For example, to switch VS Code to the Night Owl theme, the agent installs the extension via CLI, uses GUI to browse and preview the theme panel, then returns to CLI to read the config file and confirm the setting persisted.

3. Interface Selection as an Explicit Training Objective

The team synthesized 3,000 verifiable RL tasks. Each task was sampled 16 times under pure GUI, pure CLI, and mixed modes; the results labeled whether CLI usage was beneficial. Training proceeds in two stages: supervised fine-tuning on the three trajectory types, followed by online GRPO reinforcement learning in a real GUI–CLI environment with two auxiliary rewards:

Task-level reward : granted only when the task succeeds and the CLI usage matches the label — teaches "when to use CLI".

Step-level penalty : applied per shell command that errors — teaches "how to use CLI correctly".

Experimental Results

Using Qwen3.5-9B as the base model, the team fine-tuned for 2 epochs on the 5,023 trajectories, then ran RL on 1,000 sampled tasks. Evaluation covered all 361 OSWorld tasks (max 50 steps).

Headline numbers : HybridCUA-9B reaches 53.6% accuracy with an average of 14.0 steps — both best among same-size models. Compared to the base model, accuracy rises 14.8 points and steps drop from 31.6 to 14.0. On WindowsAgentArena it improves 4.0 points over the base.

Isolating the hybrid advantage : A pure-GUI control trained on identical data, same RL tasks, and same steps reaches only 50.4% accuracy and 22.1 steps. The mixed version's 53.6% / 14.0 shows the gain comes from GUI–CLI orchestration, not just more training.

Against specialized-API methods : HybridCUA-9B leads AutoGLM-OS-9B by 4.7 points, ToolCUA-8B by 6.8 points, and even surpasses the ~4× larger UltraCUA-32B by 9.9 points.

Ablation Studies

SFT with all three trajectory types yields 46.0%, outperforming pure GUI (+2.8), pure CLI (+14.3), and mixed-only (+5.0). Adding RL contributes another 7.6 points and cuts steps by 29.3%.

Removing the task-level reward leaves CLI usage at 58.9% with only 18.2% step reduction. Removing the step-level penalty lets command error rate rebound to 16.5% (vs. 11.5% for the full model). Both rewards are necessary.

Out-of-Distribution Generalization

On OSWorld-MCP (MCP-enabled), HybridCUA-9B scores 47.1% (+9.1 over base), matching ToolCUA-8B which relies on proprietary APIs. On WindowsAgentArena it reaches 36.0% (+4.0 over base, +2.2 over ToolCUA-8B). Notably, the model was trained only on Linux Shell but spontaneously uses PowerShell on Windows.

Behavioral analysis reveals sensible division of labor: 85.8% of content editing and 98.4% of result verification go to CLI, while 58.4% of positioning and layout adjustments use GUI. By domain, system tasks use CLI for 84% of steps; Chrome-centric web tasks use GUI for 74%.

HybridCUA's CLI usage ratio (64.0%) is close to Claude-Opus5's 67.0%, yet HybridCUA improves while Opus5 degrades. The transferable capability is not "use CLI more" but "judge whether a step belongs to the interface or the terminal" — a generalizable skill for building more accurate, efficient, and universal computer agents.