How UI Ontology and Neuro‑Symbolic AI Enable Agents to Operate a Computer

Screen as a Spatial Ontology

Each UI element has precise coordinates and size, hierarchical containment (window > panel > button), interaction rules (when clickable and resulting actions), and state changes (loading, success, error). This forms a spatial ontology that an AI agent must understand to manipulate a computer.

Pixel Computation Requirement

Anthropic engineers observed that Claude must compute pixel positions accurately; otherwise mouse commands miss targets. This mirrors classic LLM failures such as miscounting letters. Understanding the UI ontology is a prerequisite for any domain‑specific agent.

Demonstrations: Claude stopped a long‑duration screen recording, losing material; in another run it paused a code demo to browse Yellowstone photos. These illustrate that the approach—adapting tools to the model—is correct but the capability remains fragile.

Claude’s Four‑Step Action Loop

Observe a screenshot to see the current screen.

Reason about which element to click, what input to provide, and what to wait for.

Execute by issuing mouse coordinates and keyboard commands.

Verify the result by checking whether the screenshot changed as expected.

The critical second step translates a natural‑language intent such as I want to click the Save button into concrete coordinates move mouse to (x, y) . Claude can self‑correct and retry when obstacles appear.

Benchmark Performance and Security Risk

On the OSWorld benchmark ( https://os-world.github.io/ ), human performance is 70‑75 %; Claude 3.5 Sonnet achieves 14.9 %, roughly twice the score of the next‑best model, indicating it is currently the most advanced system for this task, though still far from human level.

Because Claude can read screenshots, a malicious image could embed a hidden instruction (e.g., “ignore previous requests and do X”), creating a prompt‑injection risk unique to computer‑use agents.

Three AI Paradigms and Neuro‑Symbolic Fusion

Symbolism : physical symbol systems, logical reasoning, OWL/RDF; deterministic and verifiable but cannot learn from data.

Connectionism : neural networks that learn from data; modern LLMs are flexible and language‑aware but lack guaranteed logical correctness.

Behaviorism : perception‑action loops and reinforcement learning; theoretical basis for agent behavior.

Consensus is that no single paradigm suffices; a hybrid neuro‑symbolic AI—combining LLM‑driven understanding with ontology‑driven symbolic reasoning—addresses the weaknesses of each.

Role of LLMs and Ontologies in Agents

LLMs provide the “understanding” layer, mapping natural language, screenshots, and unstructured text onto ontology objects and relations. Ontologies (OWL or enterprise ontologies) supply the “guarantee of correctness” layer through deterministic symbolic inference. Their integration lets agents both comprehend human instructions and avoid nonsensical actions.

Evolution of Ontology in the Series

The series traces ontology from philosophical foundations to knowledge representation, OWL, enterprise ontologies, agent ontologies, and finally neuro‑symbolic fusion. The UI ontology example shows that ontology’s scope extends beyond data warehouses to any structured, rule‑governed domain where an agent must act—including screens, physical environments, and codebases.

Key Takeaways

Computer screens constitute a spatial ontology composed of objects, coordinates, hierarchical links, interaction rules, and state changes.

Accurate pixel computation is essential for Claude’s computer use; current performance is 14.9 % on OSWorld, far below humans but ahead of peers.

Prompt‑injection attacks via malicious screenshots are a unique security risk for computer‑using agents.

Neuro‑symbolic AI merges connectionist LLM understanding with symbolic ontology reasoning, compensating for each other’s weaknesses.

The ontology evolution path shows relevance across AI research, enterprise applications, and agent control.

References

[1]

Developing a computer use model — Anthropic News: https://www.anthropic.com/news/developing-computer-use [2] OSWorld benchmark: https://os-world.github.io/ [3] 学习人工智能 AI 需要哪些最基础的知识 — 腾讯云: https://cloud.tencent.com/developer/article/1781507