Machine Learning Algorithms & Natural Language Processing

Sep 17, 2026 · Artificial Intelligence

AutoResearchExam: AI Research Agents Overfit to Validation Sets — Hidden Tests Reveal True Generalization After 24 Hours

Bespoke Labs' AutoResearchExam benchmarks nine frontier models on 29 open-ended ML research tasks for 24 hours each, revealing that validation-set gains often fail to transfer to hidden tests — exposing benchmark overfitting where models like GPT-5.6 Sol spend 55.5% of rounds on hyperparameter tuning, while Claude Fable 5.1 ultimately leads on time-weighted hidden-test AUARC despite Astra's early advantage.