BestHub
Sign in
Tagged articles

GUI automation

12 articles · Page 1 of 1
Network Intelligence Research Center (NIRC)
Network Intelligence Research Center (NIRC)
Jul 9, 2026 · Artificial Intelligence

Computer Use Explained: How Large Models Can Take Over Your Screen and Mouse

The article introduces Computer Use, a technique that lets large language models directly observe, decide, and act on graphical user interfaces—enabling them to click, type, scroll, and switch windows without relying on APIs, thereby turning AI from mere conversation into executable software actions.

AI executionGUI automationLarge Language Models
0 likes · 4 min read
Computer Use Explained: How Large Models Can Take Over Your Screen and Mouse
Smart Sea Tide
Smart Sea Tide
May 12, 2026 · Artificial Intelligence

UI-Venus-1.5: How Ant Group’s Open‑Source GUI Agent Achieves SOTA Performance

UI‑Venus‑1.5, an open‑source end‑to‑end GUI agent from Ant Group, tackles knowledge gaps, offline‑online mismatch, and high coordination costs by introducing mid‑stage training with a 10 B‑token corpus, online reinforcement learning, a “split‑then‑merge” model fusion, and a scalable DaaS layer, achieving top scores on VenusBench‑GD, ScreenSpot‑Pro and real‑world AndroidWorld tests.

AI AgentGUI automationbenchmark
0 likes · 5 min read
UI-Venus-1.5: How Ant Group’s Open‑Source GUI Agent Achieves SOTA Performance
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Mar 12, 2026 · Artificial Intelligence

LongHorizonUI: A Unified Robust Framework for Long‑Horizon GUI Agent Automation

LongHorizonUI tackles the steep success‑rate drop of GUI agents on tasks longer than 10‑15 steps by introducing three tightly coupled modules—enhanced perception, deep reflective decision, and compensatory execution—and validates the approach on the new LongGUIBench benchmark with consistent performance gains across both app and game scenarios.

GUI automationICLR 2026Multimodal LLM
0 likes · 12 min read
LongHorizonUI: A Unified Robust Framework for Long‑Horizon GUI Agent Automation
PaperAgent
PaperAgent
Dec 10, 2025 · Artificial Intelligence

How AI Agents Like UFO, Mobile-Agent, and UI-TARS Are Shaping 2025 Smartphones

The article examines the underlying GUI‑Agent technologies behind the 2025 “Doubao” smartphone, comparing Microsoft’s UFO series, Alibaba’s Mobile‑Agent v2/v3, and ByteDance’s UI‑TARS, detailing their model foundations, input modalities, action spaces, planning mechanisms, learning strategies, open‑source status, and multi‑agent frameworks.

AI agentsGUI automationMultimodal Models
0 likes · 8 min read
How AI Agents Like UFO, Mobile-Agent, and UI-TARS Are Shaping 2025 Smartphones
Network Intelligence Research Center (NIRC)
Network Intelligence Research Center (NIRC)
Nov 4, 2025 · Artificial Intelligence

SEAgent: A Self‑Evolving Computer Agent that Learns Software Use Autonomously

SEAgent introduces a self‑evolving framework that enables a GUI agent to master unfamiliar software through autonomous exploration and experience learning, leveraging a curriculum generator, a world‑state model, and GRPO‑based reinforcement with adversarial imitation, achieving state‑of‑the‑art performance on OSWorld.

Experience LearningGUI automationSEAgent
0 likes · 6 min read
SEAgent: A Self‑Evolving Computer Agent that Learns Software Use Autonomously
Xiaolong Cloud Tech Team
Xiaolong Cloud Tech Team
Oct 21, 2025 · Artificial Intelligence

Mobile-Agent-v3 and GUI-Owl: Open‑Source Breakthrough in GUI Automation

Mobile-Agent-v3 introduces the GUI-Owl multimodal agent, a self‑evolving framework that unifies perception, grounding, reasoning, planning and action execution, scales across Android, Ubuntu, macOS and Windows, leverages a novel TRPO reinforcement‑learning loop, and achieves state‑of‑the‑art results on AndroidWorld (73.3% success) and OSWorld (37.7% success) while surpassing proprietary models such as GPT‑4o with its 32B variant.

GUI automationGUI-OwlMobile-Agent-v3
0 likes · 27 min read
Mobile-Agent-v3 and GUI-Owl: Open‑Source Breakthrough in GUI Automation
Software Engineering 3.0 Era
Software Engineering 3.0 Era
Mar 14, 2025 · Artificial Intelligence

From J.A.R.V.I.S. to Real AI Agents: A Must‑Read Guide to Modern GUI Agents

This article provides a comprehensive overview of AI agents, focusing on GUI‑based agents, their definitions, classifications, core capabilities, recent research such as OpenAI's ComputerUse, SpiritSight and MobileFlow, practical applications, technical and security challenges, and future development directions.

AI agentsComputerUseGUI automation
0 likes · 16 min read
From J.A.R.V.I.S. to Real AI Agents: A Must‑Read Guide to Modern GUI Agents
ByteDance SE Lab
ByteDance SE Lab
Aug 21, 2023 · Artificial Intelligence

How Fastbot Uses Reinforcement Learning for Faster Android GUI Testing

Fastbot is a reusable, model‑based Android GUI testing tool that leverages reinforcement‑learning techniques to learn from previous test runs, accelerating coverage and crash detection through a two‑phase workflow, probabilistic and learning‑based event selection, and provides configurable custom events, widget blocking, and tree‑pruning features.

GUI automationandroid testingfastbot
0 likes · 16 min read
How Fastbot Uses Reinforcement Learning for Faster Android GUI Testing
360 Quality & Efficiency
360 Quality & Efficiency
Jun 28, 2019 · Operations

Using Sikuli for GUI Automation: Installation, Python Integration, and Practical Tips

This article introduces Sikuli, an image‑based GUI automation tool, explains its origins, provides download links, details installation steps, demonstrates Python integration via the Lackey library and SikuliX API, shares useful code snippets, and highlights common pitfalls and overall considerations for test automation.

GUI automationLackeyPython
0 likes · 6 min read
Using Sikuli for GUI Automation: Installation, Python Integration, and Practical Tips