VISTA: Visual Memory Harness Lets AI Master 25 Games with Perfect Efficiency

VISTA: A Visual Harness for Reasoning in an Interactive World

Researchers from MIT, including Kaiming He (known for MAE and MoCo), present VISTA ( V isual I nteractive S ystem for T ask A gents), a framework that equips multimodal models with a lossless visual memory to reason and act in unfamiliar interactive environments. The paper, titled "VISTA: A Visual Harness for Reasoning in an Interactive World," was authored by Qiushi Han, Keya Hu, Linlu Qiu, Cathy Wu, and Kaiming He, with the preprint released on October 1.

Problem: From Static Reasoning to Interactive Worlds

Unlike traditional visual question answering, the ARC-AGI-3 benchmark places an agent into a game without prior knowledge of objects, rules, or goals. The agent must observe, act, receive feedback, and iteratively refine its understanding. As levels progress, it must also decide whether previously discovered rules still apply.

VISTA Design: Three Core Components

VISTA departs from approaches that first convert scenes into symbolic representations. Instead, it integrates:

Visual Input: Raw PNG game frames are fed directly to the model, preserving spatial relationships and allowing on-demand zooming into local details.

Natural Language Reasoning: The model records discovered patterns, forms hypotheses, and updates beliefs based on action outcomes, enabling gradual environment understanding without a complete upfront rule set.

Lossless Visual Memory: All historical frames are retained. The model can actively retrieve any past frame, zoom into regions, or read raw pixel values. Complementary notes store reusable rules and temporary findings. When context length approaches limits, the model can summarize state and continue in a fresh context.

This combination lets the model both abstract experience via notes and verify judgments by re-examining original visual evidence.

Experimental Results on ARC-AGI-3

Using Claude Opus 5.0 and GPT-5.6 Sol as base models, VISTA was tested on the 25 public ARC-AGI-3 games (183 levels total). With Claude Opus 5.0:

All 25 games completed (100% success rate).

Average Relative Human Action Efficiency (RHAE) score of 100.00 .

Actions required reduced by 57.4% compared to first-time human players.

An ablation study examined the contribution of each component (image input mode, context capacity, visual memory, active inspection, pixel reading), showing that the full VISTA configuration consistently outperforms simpler harnesses.

Generalization to Other Benchmarks

VISTA was further evaluated on three additional benchmarks covering interactive games (GameWorld, AI GameStore) and static visual puzzles (BabyVision). Across all, VISTA significantly outperformed the same base models paired with simpler harnesses.

Limitations and Caveats

Sim-to-real gap: Real-world agents face more complex goals, diverse feedback, and less predictable state changes; VISTA's success in games does not guarantee transfer.

Base model dependence: The authors acknowledge that the underlying model's capability is a major factor in the results.

Potential data contamination: The models were released after the ARC-AGI-3 games, so exposure during training cannot be ruled out. True generalization requires evaluation on strictly unseen tasks.

Significance

VISTA demonstrates that improving agent performance does not solely rely on training larger foundation models; enhancing how models interact with environments—through direct visual access, persistent memory, and active retrieval—can yield substantial gains. This work signals a shift from static visual representation learning toward long-horizon interactive reasoning.