Skip to content
Tagged articles

deterministic evaluation

1 articles · Page 1 of 1
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Nov 19, 2026 · Artificial Intelligence

Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering – GPT-6 Astra Tops at 53% Pass Rate

Researchers from CAS, Tsinghua, PKU, and Southeast University introduce Doc2Harness, the first deterministic benchmark evaluating LLMs' ability to build coding agent harnesses from a contract skeleton and dev doc; 19 models tested, GPT-6 Astra achieves highest pass rate of 53.05% on 328 test cases, revealing significant room for improvement in long-horizon engineering.

Doc2HarnessLLM benchmarkcoding agents
0 likes · 25 min read
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering – GPT-6 Astra Tops at 53% Pass Rate