What Does DeepResearch Bench Measure? Toward Human‑Level AI Agent Evaluation

At NVIDIA GTC 2026, NVIDIA unveiled the Agent Toolkit and AI‑Q blueprint, positioning AI agents as a new frontier. To showcase AI‑Q’s deep‑research capabilities, NVIDIA highlighted the DeepResearch Bench and DeepResearch Bench II, where AI‑Q ranked first with scores of 55.95 and 54.50 respectively.

Background: the surge of deep‑research agents and evaluation challenges

Following OpenAI’s Deep Research release, companies such as Google, Grok, Perplexity, and Chinese providers (Qianwen, ByteDance’s Doubao, Tongyi) launched competing agents that can autonomously plan search paths, browse dozens to hundreds of webpages, synthesize information, and produce structured research reports in minutes. Evaluating these reports is harder than code generation or math reasoning because quality depends on completeness, depth of analysis, structure, and citation reliability, often with trade‑offs among these dimensions.

DeepResearch Bench I (ICLR 2026)

The USTC research team collected ~96 000 user queries from real interactions with search‑enhanced LLMs, filtered and anonymized them to 44 000 queries that fit the “deep‑research” definition, and identified demand distribution across 22 topic areas. From this they defined task counts and enlisted PhD‑level experts to author 100 high‑challenge research tasks (50 Chinese, 50 English).

The benchmark provides two complementary evaluation frameworks:

RACE : Dynamically generates criteria and weights per task (e.g., financial analysis emphasizes data depth, popular science emphasizes readability) and compares the agent’s report against a high‑quality reference report to avoid uniformly high scores from LLM judges.

FACT : Extracts each factual claim and its cited URL, retrieves the referenced page, and verifies whether the citation truly supports the claim, yielding two metrics – effective citations (E.Cit.) and citation accuracy (C.Acc.).

In the first evaluation round, Gemini Deep Research led in effective citation count (average 111 per task) while OpenAI Deep Research excelled in instruction following. Perplexity Deep Research achieved a 90 % citation accuracy, illustrating the distinction between “finding many” and “finding correctly”.

A human consistency experiment on 50 Chinese tasks (3 expert raters per task, 225 ratings) showed RACE’s pairwise consistency of 71.3 %, surpassing human inter‑rater agreement of 68.4 % and outperforming baseline LLM‑as‑Judge methods.

DeepResearch Bench II

Bench II addresses two fundamental issues of prior evaluation paradigms:

Pre‑defined scoring points generated by LLMs may not reflect expert priorities.

Post‑hoc citation checks verify format and accessibility but not factual correctness, risking reliance on potentially poisoned web content.

The solution is to anchor evaluation to peer‑reviewed expert reports. By reverse‑engineering these reports, the team extracted thousands of fine‑grained binary rubrics (9 430 items, ~71 per task) that answer concrete yes/no questions (e.g., “Does the report identify labor‑force outflow in small cities as caused by occupational mismatch?”). This eliminates the need for the model to judge correctness itself.

Bench II also introduces a three‑layer capability taxonomy:

Information Retrieval : Does the agent know what to look for and retrieve correct information?

Analysis : Does the agent go beyond mere summarization to generate original insights?

Presentation : Is the report organized and communicated clearly for the target audience?

These layers map to the full pipeline of “search → think → write”.

Key Findings and Outlook

The two‑generation series consistently pursues the question: “How can deep‑research agent evaluation approach human‑expert judgment?” Generation I answered by making evaluation smarter (dynamic weights, adaptive criteria). Generation II answers by making evaluation evidence‑based, using expert reports as anchors to pinpoint precise gaps between AI and humans.

Limitations acknowledged include the inherent subjectivity of research reports, possible flaws in expert articles, hallucinations during LLM extraction, and imperfect rubrics. The authors invite community feedback via an open comment section.

Future challenges highlighted are improving analysis depth and originality, and achieving user‑level adaptability (e.g., tailoring reports for undergraduates versus senior professors). All data, code, and evaluation scripts are openly available via the links provided at the article’s start.