Why Computer Use Agents Like Manus Signal a New Era for AI Automation

Computer Use Definition

In November 2024 Anthropic released a beta capability for Claude 3.5 called Computer Use . The model can perceive the screen, move the cursor, click UI elements, and type text, enabling natural‑language commands to be translated into concrete system actions without a dedicated agent per application.

Core Capabilities

Cross‑platform UI parsing – real‑time computer‑vision identification of GUI elements (buttons, input fields, menus) with reported accuracy of 92%

https://www.mittrchina.com/news/detail/13924?locale=zh_CN

Human‑like operation chain – a closed loop of screen perception → cursor positioning → click/input → result verification.

Adaptive learning framework – reinforcement‑learning‑based path optimisation that improves response speed by roughly 40% on unstructured interfaces.

Bidirectional feedback – continuous screen‑change capture during execution to dynamically adjust the action plan.

Approach 1 – Pure Vision‑Language Model (VLM)

The architecture follows a perception‑decision‑execution loop:

Perception : Dynamic screen capture streams RGB pixels at ≤100 ms latency while recording UI metadata (window hierarchy, control properties, focus).

Decision : A VLM runs object detection (Faster R‑CNN) and semantic segmentation (Mask R‑CNN) to recognise UI elements and infer user intent, then produces an action plan (mouse move, click, text entry).

Execution : System‑level input drivers carry out the generated commands until the LLM judges the task complete, a maximum step count is reached, or the context window is exceeded.

This design is conceptually simple but places high demands on the VLM’s ability to reliably recognise interactive icons and map visual regions to precise click coordinates.

Approach 2 – VLM + OCR Fusion

Adding OCR refines text understanding and positioning, extending the pipeline:

Screen capture.

Multimodal reasoning with VLM + OCR to produce structured location requests, e.g.:

[{ "reasoning": "cognitive process here", "action_type": "click", "target_text": "target element" }]

Coordinate mapping that combines visual features with OCR‑derived text positions.

Generation of a command set for the system‑level input engine.

OCR mitigates issues such as font variations, multilingual text, and decorative lettering, thereby increasing accuracy on complex interfaces.

Approach 3 – Microsoft OmniParser V2

Microsoft Research released OmniParser V2, a paradigm‑shift technology that converts any LLM into an agent capable of direct computer interaction. It parses the screen into W3C ARIA‑compliant metadata, reducing end‑to‑end latency by 60% and achieving state‑of‑the‑art precision on challenging UI benchmarks.

Dynamic screen‑state capture.

Multimodal interface parsing via OmniParser V2, outputting interactive element metadata.

Construction of VLM reasoning context.

System‑level input simulation to execute actions.

OmniParser shares the VLM + OCR foundation but delivers higher precision and speed.

Performance Observations

All three implementations exhibit performance constraints: VLM‑only approaches struggle with precise click placement; VLM + OCR improves text‑heavy UI handling but adds processing overhead; OmniParser V2 mitigates latency but still requires substantial computational resources. Current prototypes are not yet production‑ready, and efficiency remains a primary research focus.

https://www.anthropic.com/news/3-5-models-and-computer-use