From Slanted Arrow to AI Pointer: How Cursor Interaction Is Evolving

How the Cursor Became a Visual Language

In 1968 Douglas Engelbart demonstrated the mouse for the first time, showing a vertical arrow pointer that moved with the cursor on the screen, an event later called “The Mother of All Demos.”

When Xerox PARC developed the Alto graphical interface, the pointer was limited to a fixed 16×16‑pixel black‑and‑white bitmap due to display constraints.

Alan Kay discovered that a vertically upward shape became blurry at low resolution; rotating the shape so one side stayed vertical and the other slanted improved clarity. This design was adopted by Macintosh and Windows and remains the familiar cursor today.

As graphical interfaces added text editing, window resizing, and hypertext linking, a single arrow could no longer express all actions. The cursor diversified into I‑beam for text, hand shape for links, and hourglass or spinning wheel for busy states.

Custom cursors also emerged as a form of personal expression, ranging from animated cartoons to video‑game characters.

Touchscreens initially seemed to render the cursor obsolete. When the first iPhone launched in 2007, Steve Jobs declared that fingers were “the best pointing device.” The hand‑shaped cursor on desktops was replaced by the user’s actual finger on touch devices.

Cursor Adapts to Screen Content

When iPadOS introduced mouse and trackpad support, it did not reuse the traditional arrow. Instead, it created a circular pointer that morphs into an I‑beam in text fields and snaps to nearby controls, anticipating the user’s target based on speed, direction, and surrounding elements.

Cursor Becomes a Collaboration Language with AI

In 2024 Microsoft released “Click to Do,” allowing users to invoke a contextual action layer on any visible screen content. After selecting text, the system recognizes it and offers copy, summarize, rewrite, or search options; selecting an image triggers reverse‑image search or background removal; selecting an email address or URL offers sending or opening actions.

Unlike static right‑click menus, the actions depend on what the AI identifies, turning the cursor into a way of defining the current task’s context for the AI.

Microsoft’s Copilot Vision with Highlights further lets AI highlight UI elements with the cursor to guide users when they are unsure where a function resides.

In May 2026 Google DeepMind demonstrated “AI Pointer,” an experimental feature where users point at screen content and speak a command; the AI interprets intent using cursor location, screen content, and spoken words, enabling actions such as summarizing a PDF, generating a chart from a table, or booking a restaurant from a video.

Design tools are adopting similar ideas. Cursor’s Design Mode lets designers click elements, select regions, and add spoken instructions; the system records the click location, component details, style, screenshot, and surrounding layout, then passes this context to an AI agent for code modification.

These pointing actions act as spatial prompts, supplementing language with precise location, range, hierarchy, and relational information.

Conclusion

On desktop screens, the higher information density makes the cursor more suitable than fingers for precise operations. The cursor serves as the focal point of our visual attention—where we look, the cursor goes, and where we want to act, we click.

As AI agents increasingly assist with computer tasks, expressing intent clearly becomes crucial. While natural language is powerful, combining it with cursor‑based gestures provides a richer, more precise communication channel.

In this sense, the cursor is the hand we extend into the screen, allowing us to point, select, and tell AI exactly what we mean.

References

James Hill‑Khurana, “A Brief History of the Mouse Cursor, from Engelbart to PARC.” https://jameshk.com/mouse-cursor

Museums Victoria, “HyperCard Overview.” https://collections.museumsvictoria.com.au/articles/1487

Internet Archive, “Apple Macintosh System 7.0.1.” https://archive.org/details/AppleMacintoshSystem701

The Museum of Modern Art, “Susan Kare: Apple Macintosh OS icon sketchbook.” https://www.moma.org/collection/works/188382

Apple, “Design for the iPadOS pointer.” https://developer.apple.com/cn/videos/play/wwdc2020/10640/

Microsoft, “How to get new experiences for Windows 11.” https://blogs.windows.com/windowsexperience/2024/10/01/how-to-get-new-experiences-for-windows-11/

Microsoft Support, “Click to Do: Do more with what’s on your screen.” https://support.microsoft.com/zh-tw/article/6848b7d5-7fb0-4c43-b08a-443d6d3f5955

Microsoft, “Copilot Vision on Windows with Highlights.” https://www.microsoft.com/en-us/microsoft-copilot/blog/2025/06/12/copilot-vision-on-windows-with-highlights-is-now-available-in-the-u-s/

Google DeepMind, “Reimagining the mouse pointer for the AI era.” https://deepmind.google/blog/ai-pointer/

Cursor, “Direct agents with visual prompts in Design Mode.” https://cursor.com/blog/design-mode