What Core Capabilities Do Mature GUI Agents Need? Expert Insights from the Agentic AI Summit

Prof. Yang Jian (Beihang University) opened the live stream, introducing guests Zhang Xi, senior algorithm engineer at Alibaba Tongyi Lab, and Cui Chen, senior developer at Qunar AI Lab, as a pre‑event for the 2026 Agentic AI Summit in Beijing.

Core Capabilities of Mature GUI Agents

Yang asked the panel to list the core abilities a mature GUI agent should possess. Zhang identified five capabilities:

Multimodal perception and understanding, including precise UI element localization.

Planning ability to handle long‑chain tasks such as cross‑platform price comparison.

Reflection capability to detect and recover from mistakes.

World‑model knowledge of GUI actions (e.g., what happens after clicking a button).

Tool‑calling ability to invoke system APIs or scripts for higher efficiency.

Cui described a GUI agent as adding “eyes” to the traditional “brain‑and‑hand” AI agent architecture, emphasizing visual perception and the ReAct paradigm.

How Multimodal Models Understand GUIs

Yang highlighted the need for agents to map pixels to semantics. Zhang explained three challenges:

GUI images differ from natural images; they are discrete and lack the continuous features of real‑world photos, requiring extensive UI‑specific data for pre‑training.

Understanding UI controls demands domain knowledge (e.g., linking a "like" icon to the action of liking).

Element‑region‑image joint reasoning is needed because the same visual element can have different meanings across apps.

He argued that visual information is more compact than HTML/XML for token consumption, especially when source code is unavailable.

Transferring Code‑Agent Skills to GUI Tasks

Yang asked how code‑agent planning and debugging can be merged with GUI operations. Cui noted that code agents follow a Plan‑Execute (ReAct) pattern, which aligns closely with GUI agents. He suggested encapsulating actions as functions (click, swipe, etc.) to bridge the two domains.

Zhang described their current workflow: printing HTML annotations onto screenshots (a "Set‑of‑Mark" approach) to provide layout cues for multimodal models without exposing raw element coordinates.

The panel outlined a three‑level priority hierarchy for execution:

Call an available tool or function directly.

If no tool fits, generate code to accomplish the task (e.g., mkdir or Python for PPT editing).

Fallback to basic GUI actions (click, input) only when the above fail.

Edge‑Device Model Capability vs. Inference Speed

Yang asked about balancing model power and latency on resource‑constrained devices. Zhang noted typical single‑step latency of 3–4 seconds in the cloud and that current GUI agents still rely on cloud inference because on‑device models (e.g., 0.5 B parameters) are not yet sufficient.

He suggested two speed‑up strategies:

Task tiering: use small models for simple tasks and larger models for complex ones.

Cache or "combo" actions: bundle frequent sequences (e.g., search‑click‑type) into a single tool to reduce round‑trips.

Cui added that lighter models (<10 B parameters) such as a pruned version of 千问3 can achieve much lower latency, and that engineering tricks (skipping model calls when rules suffice) further cut delay.

Complex Task Planning and Error Correction

When discussing multi‑step tasks like booking a flight, Zhang proposed adding a reflection agent that receives before‑and‑after screenshots, the current action, and the user command, then decides whether the step succeeded or needs correction. Cui described a two‑layer safety net: a planning stage that splits tasks into phases, and a state‑machine fallback that can revert to a previous stable node if the agent deviates.

Tool Ecosystem, Open‑Source, and Commercialization

The panel agreed that GUI agents depend heavily on OS APIs, developer tools, and a thriving ecosystem. Zhang emphasized open‑source as a way to raise awareness, iterate through V1 (grounding), V2 (single/multi‑agent), V3 (long‑chain planning), and RL‑enhanced stages. Cui suggested that core execution frameworks stay open, while high‑value, domain‑specific fine‑tuning can be offered as a commercial service.

Deployment Challenges: Generalization, Latency, Reliability, Privacy, and Safety

Generalization is a major hurdle because UI layouts evolve rapidly; robust ViT pre‑training on diverse UI screenshots is required. Latency is mitigated by model tiering, caching, and lightweight architectures. Reliability demands multi‑step planning, error‑aware models, and rule‑based fallbacks. Privacy concerns arise from screen capture; solutions include on‑device visual encoding, encryption, or blurring sensitive regions. Safety can be enforced with whitelist/blacklist policies and user confirmations for critical actions.

Future Breakthrough Scenarios and Human‑Agent Interaction

Both experts see RPA and automated testing as the most immediate adoption paths, followed by repetitive tasks like PPT generation. They agree that GUI agents will not replace traditional UI soon; instead, they will act as assistants, with humans remaining the orchestrators issuing high‑level commands.

The discussion concluded with thanks to the speakers and an invitation to attend the Agentic AI Summit for deeper insights.