Top AI Models Score Only 39% on Real-World Deep Search Benchmark

Researchers from Hong Kong University of Science and Technology (Guangzhou), Fudan University, University of Washington, Imperial College London, and Stanford University present OpenDS (Open-Web Deep Search) , a new task paradigm for evaluating AI agents' ability to conduct autonomous, multi-step research on the live, open web. They also release OpenDS-World , a benchmark of 200 fact-seeking questions constructed by first building verifiable evidence chains from real web sources — starting from a video anchor, expanding through heterogeneous sources (text, images, audio, video; HTML, PDF, Parquet, CSV, JSON, ZIP, etc.; interactive pages requiring parameter selection, time navigation, file download) — and then inverting the chain into a question that hides intermediate entities.

Benchmark Construction

Each task begins with a Real-World Anchor : a video from which the initial entity must be identified purely from audio-visual content (not metadata). Annotators then perform Open-Web Evidence Expansion , using Google Search to find and cross-verify sources across the live web, ensuring each hop has multiple independent citations. Finally, Question Inversion writes the question so that the answer can only be reached by retracing the evidence chain. Ten independent reviewers validated evidence validity and task validity, yielding 200 tasks with an average of 3.94 annotated source entries per task. Statistics show 82.5% of tasks involve ≥2 modalities, 96.0% ≥2 formats, 95.5% ≥2 access types; 67% require page navigation, 60% resource download, 47.5% temporal localization, 42% parameterized state updates.

Human Baseline

Fifty participants (no LLM tools, only browser/search) attempted 20 tasks each (5 attempts per task, 1,000 total). They spent ~646 hours but produced only 211 correct answers; 115 tasks were solved by at least one person, 85 by none, and only 29 by ≥3 people. Tasks were tiered: Low (≥2 solvers, 54 tasks), Medium (exactly 1 solver, 61), High (0 solvers, 85).

Model Evaluation

Twelve frontier models (GPT-6 Astra, GPT-5.6 Sol, Claude Opus 5.5, Gemini 3.8 Flash, Qwen3.8 Max, Qwen3.8-Flash, GLM-5.3 Flash, DeepSeek V4.1-Flash, Kimi K3, etc.) were given identical toolsets: 16 tools covering search, browser GUI interaction (click, input, scroll, screenshot, download), and video/subtitle inspection. Each task allowed 3,600 seconds and 100 tool calls; upon budget exhaustion, models entered a no-tool answer phase. Answers were judged for semantic equivalence by Qwen3.8-27B.

Overall Accuracy

GPT-6 Astra led at 39.0%, followed by Claude Opus 5.5 (38.0%) and Gemini 3.8 Flash (36.5%); remaining models ranged 12.0%–33.5%. Accuracy dropped sharply with difficulty: average across models was 39.66% (Low), 23.50% (Medium), 16.76% (High). Gemini 3.8 Flash excelled on High (28.24%).

Search Trajectory vs. Evidence Chain

Despite only ~4 annotated sources per task, model trajectories contained dozens to nearly 100 tool calls (median 74 even for 3-source tasks). Execution length did not scale monotonically with evidence-chain length (3 sources: 74 calls; 4: 56; 5: 72; ≥6: 82.5), highlighting that long-horizon decision-making — discovering, accessing, and correlating scattered evidence — dominates cost, not chain length alone.

Tool Usage Patterns

Models differed markedly: GPT-5.6 Sol used search for 74.5% of calls; Claude Opus 5.5 and Qwen3.8-Flash relied on GUI (51.5%, 49.4%) but with a 26-point accuracy gap. High self-transition rates appeared (Gemini Search→Search 84.6%, Qwen GUI→GUI 85%), suggesting limited adaptive tool coordination. The paper concludes that raw call counts and category distributions do not explain performance differences ; effective orchestration of search, interaction, and evidence integration is key.

Failure Call Overhead

Failure rates ranged 2.83%–9.42% of calls but consumed 19.27%–61.90% of total tool execution time. For GPT-6 Astra, 6.34% failed calls ate 42.57% of time. Most models reacted to failure by switching tools (Claude 83%, Gemini 81%, GPT-5.6 Sol 79%), while Kimi K3 (38%) and GLM-5.3 (37%) preferred retrying with modified parameters. Reducing futile operations and improving failure recovery emerge as critical optimization directions.

Search Termination

GPT-6 Astra naturally stopped (submitted answer without further tool calls) on 69% of tasks, yet its accuracy was only 39% — voluntary termination ≠ sufficient evidence . Other models frequently hit budgets: Qwen3.8-Flash 80% call-limit, DeepSeek V4.1-Flash 60.5%, GLM-5.3 41% time-limit. Even models with low budget-exhaustion rates stayed below 40% accuracy, underscoring dual challenges: completing search within budget and judging when evidence is adequate .

Future Direction: Recursive Self-Improvement

The authors propose OpenDS-World as a natural environment for Recursive Self-Improvement (RSI) : agents can analyze their own execution traces (tool calls, feedback, failures, resource use) to refine source selection, tool coordination, recovery strategies, and stopping criteria, then validate improvements on new tasks. Because live web content, page structures, and resource states constantly change, agents must develop adaptive research strategies grounded in real interaction feedback.

Resources: Paper https://arxiv.org/pdf/2610.30181v1, GitHub https://github.com/Yu-xm/OpenDS, Hugging Face dataset https://huggingface.co/datasets/Yu2020/OpenDS-World.