The Real Bottleneck for ML Agents: Choosing Experiments, Not Coding (6× Faster)

The paper "Can We Predict Before Executing Machine Learning Agents?"—awarded an ACL 2026 SAC Highlight and authored by researchers from Zhejiang University and Ant Group—investigates why modern AI agents that can write and run ML experiments still struggle with efficiency.

Current ML agents follow a Generate‑Execute‑Feedback loop: they generate many candidate solutions, execute each full training pipeline (pre‑processing, training, evaluation), and then learn from the results. Because a single execution on benchmarks such as MLE‑Bench can take several hours, agents are limited by the physical cost of the "execute" step, often being able to run only one out of ten generated candidates.

To address this, the authors formalize the Data‑centric Solution Preference task: given a task description, a verified data‑analysis report, and two candidate code snippets, the model must predict which solution is more likely to succeed and provide a confidence score. They construct a large Preference Corpus from two real ML agents (AIDE and AutoMind) covering 26 CV, NLP, and Data‑Science tasks, yielding 895 high‑quality instances and 18,438 pairwise comparisons.

Experiments show that large language models can indeed make useful pre‑execution predictions. DeepSeek‑V3.2 in Thinking mode achieves 61.5% average accuracy, GPT‑5.1 reaches 58.8%, both far above the random baseline (50.0%) and a complexity‑based heuristic (50.8%). Adding data context improves performance, and converting raw statistics into a semantic Verified Data Report yields the best results. The authors further analyze factors such as data representation, reasoning mode (Thinking/CoT vs. direct answer), and model scale, noting that performance plateaus after ~30B parameters.

Building on these findings, the paper proposes FOREAGENT , a Predict‑then‑Verify architecture. FOREAGENT generates multiple candidates in parallel (m=10), uses the LLM’s confidence (threshold c=0.7) to gate them, and only executes the top‑k (k=1) proposals. On five AI‑for‑Science tasks from MLE‑Bench, FOREAGENT delivers an average 6× speed‑up, explores 3.2× more nodes, and improves the Beat Ratio by 6% compared with a traditional Execute‑then‑Learn agent.

The authors argue that pre‑execution prediction shifts ML agents from a purely execution‑driven feedback loop to a hybrid one where internal world‑model reasoning filters low‑value experiments, a capability crucial for costly AutoML and scientific discovery pipelines. Limitations include the current focus on pairwise comparisons (global ranking accuracy drops to 31.1%), a modest 61.5% prediction accuracy, and a validation‑test gap of 72.2%, indicating room for richer multimodal data reports and more advanced reasoning strategies.