MLS‑Bench: A New Benchmark That Strips Away the Illusion of AI Research Gains

The paper presents MLS‑Bench, a collaborative benchmark created by 28 researchers from ten universities—including UC Berkeley, Purdue, and Tsinghua—to evaluate whether AI agents truly discover new methods or merely exploit engineering optimizations.

Existing Auto‑Research evaluations suffer from attribution ambiguity: improvements may stem from new optimizers, data handling tweaks, or hidden loopholes in the evaluation pipeline, making it hard to separate methodological innovation from engineering gains.

MLS‑Bench addresses this by defining 140 real‑world research tasks drawn from twelve directions (LLM pre‑training, vision generation, RL, AI‑for‑Science, etc.). Each task includes (1) a concrete research problem with its codebase, (2) a calibrated modification scope, (3) at least three test conditions to assess transferability, (4) three strong human baselines (including SOTA), and (5) reproducible baseline results within the same repository and training/evaluation protocol.

Evaluation uses a three‑level scoring scheme: individual metrics are anchored to human baselines (0–50 points), aggregated across test conditions, then summed across tasks. This design ensures that a higher total score reflects consistent improvement across diverse settings rather than a single‑instance boost.

In the main experiment, five frontier models were tested in two modes: (a) submitting a single initial proposal, and (b) iteratively improving the solution over multiple rounds. Multi‑round runs raised overall scores, yet none of the models surpassed the Human SOTA on the full benchmark. Ablation studies showed that gains were largely due to local parameter adjustments, component recombination, or increased model capacity rather than novel algorithmic ideas.

Further analysis revealed a tension between openness and attribution: allowing unrestricted repository edits makes it impossible to pinpoint the source of improvement, while overly tight modification limits can suppress genuine innovation. MLS‑Bench calibrates this balance by fixing data pipelines, training budgets, and evaluation protocols, and by enforcing capacity constraints during experiments. When the capacity check is removed, models artificially inflate scores by simply scaling up parameters, confirming the necessity of strict controls.

Additional experiments varied the number of iterations and sampling budget. More attempts improve simple tasks but quickly saturate, and do not alter the fundamental behavior of agents, which continue to explore a predefined candidate space rather than invent new mechanisms.

A budget‑constrained pre‑training study gave agents freedom to allocate resources across cheap hypothesis tests and expensive full‑scale training. Greater freedom did not consistently yield better results; only one model improved by using a minimal budget, while others wasted resources and performed worse.

The authors conclude that current large‑scale search excels at engineering optimization—optimizing and recombining known components—but falls short of true method discovery, which requires hypotheses that generalize across data, models, and scales. MLS‑Bench therefore aims to push AI research toward scalable scientific discovery under limited resources.

Adoption of MLS‑Bench‑Lite by Kimi K3 and Qwen 3.8‑Max demonstrates growing interest, and future work should focus on agents that can formulate insightful hypotheses, select informative experiments, and validate findings across multiple conditions.