Strong Model ≠ Strong Agent: PolyWorkBench Benchmarks Cross-Lingual Long-Horizon Workflows

Large language model (LLM) agents have evolved from simple Q&A to complex long-horizon tasks involving planning, tool use, file operations, and multi-turn interactions. However, real-world work often requires continuous cross-lingual understanding, planning, tool calling, information integration, and result generation within a single execution trajectory — not just at input and output boundaries.

To evaluate agents in such scenarios, researchers from Beijing Jiaotong University and Tencent WeChat AI introduced PolyWorkBench , a benchmark for cross-lingual long-horizon workflows. The benchmark comprises 67 manually designed tasks sourced from real work scenarios, audited by translation and computer professionals, covering five domains: Commerce (COM), Knowledge (KNW), Legal (LEG), Localization (LOC), and Manufacturing (MFG) . Tasks involve cross-border business, market analysis, contract review, software localization, quality management, and production reports across 10 languages : English, Chinese, Japanese, Korean, French, German, Spanish, Russian, Vietnamese, and Arabic.

Each task requires the agent to iterate through heterogeneous resources and tools — multilingual documents, spreadsheets, databases, browsers, and code editors — following a full workflow: understand task → retrieve information → multilingual comprehension → plan → call tools → integrate information → generate results . On average, a task includes 3.4 input files and 2.3 languages , with multiple structured scoring criteria for the final output.

Three-Layer Evaluation System

PolyWorkBench avoids a single LLM judge and instead uses three complementary signals:

Grade : structured rule-based evaluation of task completion (primary ranking metric).

Pytest : automated testing of executable artifacts.

LLM-as-Judge : semantic quality and natural language fluency assessment.

This design allows multiple valid execution paths and answers three questions: task completion, structural/executable correctness, and semantic reasonableness.

Experimental Results

The team tested multiple model–harness combinations on the 67 tasks. The best-performing combination was Claude Opus 4.8 + ClaudeCode achieving Pass@1 = 92.3% , followed by GLM-5.2 + Codex at 88.7% .

A critical finding: the same model with different agent harnesses yields drastically different performance . For Claude Opus 4.8, Pass@1 across four harnesses ranged from 92.3% down to 69.8% — a 22.5 percentage-point gap . This demonstrates that final agent capability depends not only on the base model but also on the planning, tool-use, and execution framework.

Domain and Language Analysis

Domain variation: Strong models scored 0.85–0.95 Grade on Knowledge, Legal, and Manufacturing tasks, but performance dropped noticeably on Commerce tasks. Commerce tasks typically combine numerical verification, cross-currency calculations, spreadsheet operations, and strict format constraints , where a single local error cascades into total failure.

Language variation: Stronger agents showed relatively stable performance across all 10 languages. Mid-tier models, however, exhibited significant drops on Russian (RU), Spanish (ES), and German (DE), with the gap between best and worst language exceeding 30 Grade points for a fixed model.

Failure Modes

Analysis identified two primary failure categories in multilingual settings:

Language understanding errors — the agent misreads numbers, entities, or other key information from source-language materials and propagates the error downstream.

Cross-lingual coordination errors — the agent initially understands correctly, but as multi-turn tool calls and information integration proceed, source-language information drifts relative to the target-language output.

These failure modes are difficult to capture in traditional multilingual QA benchmarks, which typically keep each task monolingual.

Conclusion

PolyWorkBench places LLM agents into realistic, cross-lingual, multi-tool, long-horizon workflows. Results show that model capability, agent harness, task domain, and language all materially affect work completion quality . The benchmark underscores that a strong model does not guarantee a strong agent ; the synergy between model and harness is decisive. Future agent evaluation must move beyond "Can it answer this question?" to "Can it complete a complex, multilingual, multi-tool task end-to-end?"