Only 33% of Claude, GPT, and Gemini Survive Real‑World Websites—ClawBench Shows AI Agents Still Struggle

Background and Motivation

Existing web‑agent benchmarks such as WebArena, OSWorld, and Mind2Web evaluate agents in sandboxed or static environments, giving scores of 65‑75 % and suggesting that agents are close to handling real tasks.

ClawBench Design

ClawBench (paper “ClawBench: Can AI Agents Complete Everyday Online Tasks?”) expands evaluation to 144 production websites (e.g., Google Flights, DoorDash, Zillow, Airbnb, LinkedIn, Doodle) covering 15 everyday categories and 153 tasks that require write‑heavy operations such as ordering, booking, and form submission.

To prevent agents from causing real‑world effects, the benchmark uses a “final‑request interception” mechanism: human annotators identify the irreversible HTTP request for each task, and a Chrome extension with CDP monitors outbound traffic, dropping the final request while recording it. The paper reports 100 % capture of these requests in 153 human reference runs with zero false positives.

Key Findings

Overall success rate : 33.3 % of tasks are completed correctly across all models.

Claude Sonnet 4.6 : completes roughly two‑thirds of tasks; fails on 44.4 % of tasks where no model succeeds.

GPT‑5.4 : 6.5 % success (10 / 153 tasks); average 13 tool calls per task, 85 tasks exited early, median runtime 196 s.

Qwen 3.5 : 26.1 % success with $1.02 API cost per task, median 42 tool calls.

GLM‑5 : 24.2 % success, cheapest at $0.64 per task, but 48.4 % of tasks time‑out (30 min limit).

Failure Modes

Two recurring problems dominate the error analysis:

Last‑mile non‑commit – agents reach the final confirmation page but do not issue the irreversible request (e.g., Claude Sonnet stops halfway through address entry on Soko Glam).

Anti‑bot defenses – Cloudflare Turnstile, PerimeterX, DataDome, etc., block automated browsers, causing agents to retry endlessly or abort (e.g., GLM‑5 on Zillow).

Failed trajectories also involve higher operation counts than successful ones (Claude Sonnet median 120 vs 64 operations; Gemini 3 Flash median 74 vs 45), indicating agents waste tokens looping on captchas, form validation, or anti‑scraping challenges.

Domain‑Specific Performance

Shopping: Claude Sonnet 62 %.

Social: Claude Sonnet 75 %.

Work: GLM‑5 38.1 % (Sonnet third).

Dev & Tech: Haiku 4.5 and Gemini 3 Flash 27.8 %.

Travel: Qwen 3.5 and Gemini 3 Flash top.

Job Search: all models near 0 %.

Automation: 0 % for Sonnet.

These results show that current web agents are highly domain‑specialized rather than generally capable.

Comparison with ProgramBench

ProgramBench (SWE‑Bench team) asks whether AI can build software from scratch; all models scored 0 %.

ClawBench asks whether AI can act as a personal assistant on real sites; the best models still achieve only about one‑third success. Together they illustrate that while LLMs excel in controlled code generation and sandbox navigation, they lack robustness for long‑horizon, write‑heavy, real‑world interactions.

Implications

The gap is not merely “more data or compute”; it stems from interaction dynamics, anti‑bot detection, and the inability to safely execute irreversible actions. Open‑source models can be cost‑effective (Qwen 3.5, GLM‑5) but share the same fundamental limitations.

Conclusion

ClawBench provides the first systematic measurement of how far AI agents have traveled from laboratory settings to the live internet. Future progress will require advances in human‑like interaction dynamics, reliable final‑request handling, and robust anti‑bot strategies.