ACL 2026 Best Resource Paper Reveals AI Agents’ Expert-Level Capability Gap

Background and Motivation

Deep Research Agents have become a focal point for large‑model deployment, but their performance on real‑world expert tasks remains unclear. The authors examine the task of assigning the correct HS Code to a product, a 10‑digit hierarchical classification that underpins ~USD 26 trillion of global trade.

Hierarchical Rule Application

In customs clearance, experts follow a rule tree that descends from chapter (2 digits) to heading (4 digits), sub‑heading (6 digits) and finally the 8/10‑digit country‑specific code. A single mistake at any level invalidates the entire code. The authors formalize this as a “Hierarchical Rule Application” problem.

HSCodeComp Benchmark

HSCodeComp is the first benchmark that targets realistic, expert‑level scenarios. It contains 632 real products covering 32 categories, annotated by 26 customs experts (average >5 years experience) with a disagreement rate of ~2 %. The dataset includes multimodal product records (title, attributes, images, price), a layered rule set (official tariff rules + expert decision rules), and a knowledge base of historical customs rulings (CROSS).

Exact‑Match Accuracy is measured at each hierarchy depth (2, 4, 6, 8, 10 digits).

Evaluation of State‑of‑the‑Art Agents

The authors evaluated 28 systems, including GPT‑5.5, Claude‑Opus‑4.8, GLM‑5.2, DeepSeek‑V4‑Pro, Qwen‑3.7‑Max and open‑source frameworks such as Hermes‑Agent, SmolAgents, AWorld, AgentOrchestra, OWL, WebSailor. The best agent achieved only 49.4 % Exact‑Match on the full 10‑digit code, far below the 95 % of human experts.

Performance drops sharply as the rule depth increases.

Test‑time scaling strategies (majority voting, self‑reflection) provide negligible or negative gains.

Reasoning drift occurs when agents “think more”; GPT‑5’s 10‑digit accuracy falls from 40.82 % to 35.44 % with increased reasoning steps.

Key Findings

Three factors drive performance:

Rule/knowledge retrieval (+8.5 pts) – integrating an agent scaffold that fetches the latest expert rules raises GPT‑5’s 10‑digit accuracy from 28.96 % to 37.42 %.

CROSS historical ruling database (+5‑10 pts) – providing few‑shot precedent cases consistently improves all backbone models.

Visual information (+4 pts) – only strong vision‑language models (e.g., GPT‑5) benefit; weaker models see little gain.

The results indicate that success hinges on accurate, step‑by‑step rule application rather than raw model size or compute.

Data Difficulty Analysis

Difficulty concentrates in long‑tail categories such as “Novelty & Special Use” and “Men’s Clothing”. Most categories have average 10‑digit accuracy below 25 %, showing that hierarchical rule application is intrinsically hard regardless of frequency.

From Benchmark to Production

Using insights from HSCodeComp, the team built a Qwen‑based agent pipeline (multimodal parsing → retrieval‑augmented deep search → tool‑driven stepwise verification → auditable output). After SFT + Agentic RL on the 6 k expert annotations, the system reaches ~65 % accuracy, outperforming the best generic agents.

Nevertheless, a gap to the 95 % human level remains, highlighting the need for error back‑tracking, dynamic rule prioritization, and tighter anchoring of reasoning to expert knowledge.

Related Work in the Marco‑DeepResearch Series

Other releases from the ATH‑MaaS team include UMEM (ICML 2026), DeepWideSearch, Table‑as‑Search (ACL 2026) and Marco DeepResearch, all open‑sourced on GitHub.

Conclusion

HSCodeComp demonstrates a structural bottleneck for AI agents: mastering hierarchical rule application cannot be solved by scaling alone. A reproducible diagnostic‑optimisation loop—benchmark → identify gaps → augment retrieval and rule use → iterate—offers a path forward for many high‑value verticals such as medical coding, legal compliance and tax audit.