Tagged articles
GUI automation
14 articles · Page 1 of 1
DeepSeek Harness publishes its six-direction product roadmap adding remote and mobile access, personalized long-term memory, browser and GUI automation, agent team collaboration, session sharing for multi-user workflows, and enhanced sandbox security, while revealing it leads as the most-used coding agent among DeepSeek API users.
AI coding agentDeepSeek HarnessGUI automation
0 likes · 9 min read

Researchers from PhAI Labs, CUHK, and University of Macau open-sourced RSI-Anything, a framework that combines world model predictions with real-environment feedback to enable recursive self-improvement of AI agents across code repair, GUI automation, lab planning, chemical optimization, and immunology research.
AI agentsGUI automationRSI-Anything
0 likes · 8 min read

The article introduces Computer Use, a technique that lets large language models directly observe, decide, and act on graphical user interfaces—enabling them to click, type, scroll, and switch windows without relying on APIs, thereby turning AI from mere conversation into executable software actions.
AI executionGUI automationLarge Language Models
0 likes · 4 min read

UI‑Venus‑1.5, an open‑source end‑to‑end GUI agent from Ant Group, tackles knowledge gaps, offline‑online mismatch, and high coordination costs by introducing mid‑stage training with a 10 B‑token corpus, online reinforcement learning, a “split‑then‑merge” model fusion, and a scalable DaaS layer, achieving top scores on VenusBench‑GD, ScreenSpot‑Pro and real‑world AndroidWorld tests.
AI AgentGUI automationbenchmark
0 likes · 5 min read

LongHorizonUI tackles the steep success‑rate drop of GUI agents on tasks longer than 10‑15 steps by introducing three tightly coupled modules—enhanced perception, deep reflective decision, and compensatory execution—and validates the approach on the new LongGUIBench benchmark with consistent performance gains across both app and game scenarios.
GUI automationICLR 2026Long-Horizon Tasks
0 likes · 12 min read

The article examines the underlying GUI‑Agent technologies behind the 2025 “Doubao” smartphone, comparing Microsoft’s UFO series, Alibaba’s Mobile‑Agent v2/v3, and ByteDance’s UI‑TARS, detailing their model foundations, input modalities, action spaces, planning mechanisms, learning strategies, open‑source status, and multi‑agent frameworks.
AI agentsGUI automationMultimodal Models
0 likes · 8 min read

SEAgent introduces a self‑evolving framework that enables a GUI agent to master unfamiliar software through autonomous exploration and experience learning, leveraging a curriculum generator, a world‑state model, and GRPO‑based reinforcement with adversarial imitation, achieving state‑of‑the‑art performance on OSWorld.
Experience LearningGUI automationSEAgent
0 likes · 6 min read

This article surveys the evolution of GUI automation from rule‑based scripts to multimodal large‑model‑driven agents, detailing core architectures, key components, application scenarios, current challenges, and future research directions for intelligent GUI agents.
GUI automationHuman-Computer InteractionIndustry Survey
0 likes · 19 min read

Mobile-Agent-v3 introduces the GUI-Owl multimodal agent, a self‑evolving framework that unifies perception, grounding, reasoning, planning and action execution, scales across Android, Ubuntu, macOS and Windows, leverages a novel TRPO reinforcement‑learning loop, and achieves state‑of‑the‑art results on AndroidWorld (73.3% success) and OSWorld (37.7% success) while surpassing proprietary models such as GPT‑4o with its 32B variant.
GUI automationGUI-OwlMobile-Agent-v3
0 likes · 27 min read

This article provides a comprehensive overview of AI agents, focusing on GUI‑based agents, their definitions, classifications, core capabilities, recent research such as OpenAI's ComputerUse, SpiritSight and MobileFlow, practical applications, technical and security challenges, and future development directions.
AI agentsComputerUseGUI automation
0 likes · 16 min read

This article introduces the pywinauto library, explains how to install it, create Application objects, locate windows and controls, and perform mouse and keyboard automation on Windows applications using Python, with detailed code examples and usage tips.
GUI automationKeyboardMouse
0 likes · 8 min read

Fastbot is a reusable, model‑based Android GUI testing tool that leverages reinforcement‑learning techniques to learn from previous test runs, accelerating coverage and crash detection through a two‑phase workflow, probabilistic and learning‑based event selection, and provides configurable custom events, widget blocking, and tree‑pruning features.
GUI automationandroid testingfastbot
0 likes · 16 min read

This article introduces PyAutoGUI, a cross‑platform Python library for automating mouse and keyboard actions, explains its coordinate system, outlines key functions for mouse, keyboard, dialogs, and screenshots, and provides practical code examples for drawing and image‑based button clicking.
GUI automationPythoncross-platform
0 likes · 5 min read

This article introduces Sikuli, an image‑based GUI automation tool, explains its origins, provides download links, details installation steps, demonstrates Python integration via the Lackey library and SikuliX API, shares useful code snippets, and highlights common pitfalls and overall considerations for test automation.
GUI automationLackeyPython
0 likes · 6 min read
