Why Cutting‑Edge LLMs Still Miss the Mark: OmniaBench’s 644 Hard Tasks Expose Gaps

Motivation for a New General Agent Benchmark

A truly "general" AI agent should do more than answer questions in a static dialog; it must understand ambiguous user intent, retrieve scattered information, invoke multiple tools, maintain changing environment state, handle files and spreadsheets, write and execute code, and obey constraints over long chains of actions.

Existing benchmarks focus on single‑turn, fixed‑scenario tasks and cannot assess whether a model can reliably complete real‑world tasks across consumer, business, or employee contexts.

OmniaBench Overview

OmniaBench was created by Huawei Cloud’s post‑training team together with Peking University’s DCAI group to provide a benchmark that emphasizes executable environments and fine‑grained diagnostics. It covers ToC, ToB, and ToE scenarios, organizing them into a hierarchical taxonomy of 90 primary domains and 354 secondary domains .

From this taxonomy, the benchmark constructs 1,431 complete tasks . To keep evaluation cost manageable and avoid data‑pollution risk, a public challenge set of 644 high‑difficulty tasks is released for ranking.

Construction Process

Domain Knowledge Extraction : For each secondary domain, a Web Agent gathers real‑world applications, product functions, business processes, and user needs. Human curators then align concepts, boundaries, and constraints.

Executable Environment Generation : The collected knowledge is translated into Python environment classes that define entities, attributes, state variables, and domain constraints, together with tool schemas and initialization configs.

Environment Validation : Syntax checks, tool‑call roll‑outs, and state‑consistency verification filter out unstable environments.

Task Generation : Using four complementary routes (DAG, DAG‑S, Solver, Program), the system samples executable tool chains, creates natural‑language requests, reference trajectories, and target final states, followed by consistency checks.

Evaluation Signal Construction : DAG, DAG‑S, and Solver tasks receive multi‑rubric scores (1–3 weighted points) covering overall quality and task‑specific constraints; Program tasks use a binary VerifyCode check.

Task Routes

DAG (354 tasks) : Multi‑round, stateful interactions with tool dependency graphs.

DAG‑S (200 tasks) : Single‑round version of DAG tasks after query rewriting.

Solver (60 tasks) : Focus on selection, scheduling, allocation, and optimization under multiple constraints.

Program (30 tasks) : Emphasize program synthesis, branching, iteration, execution, and debugging.

Capability Profiling

OmniaBench defines a ten‑dimensional capability space: task understanding, information retrieval, planning & decision‑making, tool use, state management, office/document processing, data analysis, code & programmatic operation, interaction collaboration, and reliability & safety. It also characterizes eight atomic difficulty factors for each task.

Model Evaluation

Twenty‑two representative closed‑source and open‑source models were evaluated. The top overall Pass@1 scores were:

Claude‑Sonnet‑5: 58.54%

GPT‑5.6‑Sol: 57.14%

GLM‑5.2: 56.83%

No model surpassed 60% overall, meaning more than four out of ten hard tasks still fail for the best systems.

Analysis of route‑wise scores shows divergent strengths: GPT‑5.6‑Sol excels on Solver (65.00) but lags on Program (50.00); GLM‑5.2 dominates DAG‑S (69.00) yet scores only 26.67 on Solver. Qwen3.7‑Max achieves the highest Program score (66.67) but has a lower overall (49.69), illustrating that overall rank hides nuanced capability differences.

Efficiency and Reliability

Although Claude‑Sonnet‑5 and GLM‑5.2 have similar overall success rates, their average ToolSteps differ markedly (38.73 vs. 57.08), indicating that the latter takes many redundant or ineffective steps.

Repeated runs show Pass@k improves with more attempts, but the stricter Pass^k (requiring success on every run) drops sharply, highlighting the gap between occasional correct paths and stable reproducibility.

User Simulator Impact

Three user simulators (GPT‑4.1, DeepSeek‑V4‑Flash, DeepSeek‑V4‑Pro) were tested. Relative model rankings remained unchanged, with an average coefficient of variation of 7.24%, confirming that the benchmark is not overly dependent on a single simulator.

Error Attribution

Fine‑grained analysis of Claude‑Opus‑4.7 failures shows that 53.8% of errors stem from reasoning, of which planning & task decomposition account for 36.7% and constraint violations 16.5%. Meta‑cognitive errors (e.g., insufficient reflection, premature termination) contribute 31.0%.

Tool‑format errors, data‑dependency failures, and redundant calls are comparatively minor, indicating that the main bottleneck is long‑term planning and constraint maintenance.

Key Takeaways

Overall scores alone are insufficient; multi‑dimensional capability, scenario, and efficiency analyses are essential.

Partial completions must be visible; rubric‑based signals expose where models succeed or fail.

Action efficiency (fewer, more purposeful steps) is a critical aspect of general‑agent ability.

Higher Pass@k does not guarantee reliability; stable Pass^k is more relevant for production.

Future improvements should shift from tool‑format correctness to closing the cognitive planning loop.

Open‑Source Resources

Project homepage: https://scuuy.github.io/OmniaBench/ Paper: https://arxiv.org/abs/2607.14989 GitHub repository: https://github.com/scuuy/OmniaBench Dataset: https://huggingface.co/datasets/scuuy666/OmniaBench

The repository provides an Apache‑2.0‑licensed evaluation framework supporting OpenAI‑compatible and Anthropic‑native APIs, with scripts to download the challenge set from Hugging Face, enabling reproducible experiments and easy integration of new agents.