Which LLM Is the True AI Software Engineer? Introducing the First Living Visual Spec‑to‑App Benchmark

Recent advances in coding agents such as Claude Code, OpenAI Codex, Cursor, and Gemini CLI have enabled large language models to generate, debug, run, and even deploy full web applications, turning the notion of an “AI software engineer” into a realistic prospect. However, existing benchmarks like SWE‑bench and OpenHands focus on GitHub issue resolution rather than full product development.

To address this gap, researchers from the University of Arizona, Zoom, and Stony Brook University introduced VISTA (Visual Spec‑to‑App Benchmark), the first end‑to‑end benchmark that requires agents to start from a product requirements document and a Figma design and produce a complete, interactive web app. Unlike traditional benchmarks that ask agents to fix existing code, VISTA evaluates the entire development workflow.

VISTA pursues three goals: (1) it mirrors real software development by demanding page navigation, state management, interaction logic, and deployment; (2) it drives development with visual specifications, providing both screenshots and structured Figma JSON as ground truth; and (3) it is designed as a living benchmark, continuously updated as new models and harnesses appear.

The benchmark covers ten typical web‑app categories (news, real‑estate, recruitment, forum, travel booking, chat, cloud storage, e‑commerce, project management, music streaming), comprising 128 pages, 3,253 interactive components, and 458 visual anchors. To avoid data contamination from publicly available web pages, VISTA starts from Figma designs, retaining layout, component hierarchy, text labels, and interaction targets while discarding unrelated code.

Evaluation follows a DOM‑Grounded approach with four steps: (1) coordinate alignment using high‑confidence semantic anchors; (2) DOM element matching in a real browser to verify structural consistency; (3) behavior checks for navigation, state changes, and backend updates; and (4) aggregation of localization and behavior scores, where the final structure‑and‑function score is the average of the product of the two per‑interaction scores.

Leaderboard analysis shows three emerging trends: (a) competition has shifted from pure model performance to a system‑level contest of model + harness; (b) the gap between leading models (fable‑5, Claude Opus 4.8, GPT‑5.5, GLM‑5.2) is narrowing, yet the highest composite score remains below 0.3, indicating substantial room for improvement; (c) the “best” model is not necessarily the fastest or cheapest—fable‑5 achieves the top quality but consumes ~750 k tokens per task, while GLM‑5.2 uses ~300 k tokens and GPT‑5.5 around ~280 k tokens, with corresponding differences in wall‑clock time.

Beyond final scores, VISTA decomposes each development run into four phases—Inspect, Write, Verify, and Failure Recovery—to compare workflow styles. Claude‑series agents tend to perform repeated context checks and extensive error diagnosis, whereas GPT‑series agents switch more frequently between verification and recovery, reflecting distinct engineering strategies.

VISTA is released as an open‑source project (paper on arXiv, code on GitHub, online leaderboard) and will be continuously refreshed as newer LLMs and harnesses are integrated, providing the community with a reproducible, evolving platform for assessing AI‑driven software engineering.

Overall, VISTA advocates moving software‑engineering evaluation from code‑centric to product‑centric metrics, measuring quality, speed, cost, and workflow to answer the fundamental question: can an AI truly deliver a usable software product?