ML Agents' Real Bottleneck: Choosing Experiments, Not Coding – 6× Faster Search

Background

Modern AI agents can generate code, run experiments, and iterate, but the Generate‑Execute‑Feedback loop is limited by the expensive execution phase, which may take hours per machine‑learning pipeline.

Task Definition: Data‑centric Solution Preference

The authors formalize a task where, given a machine‑learning task description, a verified data‑analysis report, and two candidate solution codes, the model must predict which solution will perform better and output a confidence score.

Verified Data Analysis Report

A three‑step report is generated to provide the model with useful data context:

Profiling – an LLM writes code to extract statistics such as data size, distribution, missing values, image resolution, text length, etc.

Verification – experts check the profiling logs to ensure reliability.

Verbalization – raw numbers are transformed into a structured, semantic report that resembles human modeling experience.

This report bridges the gap between raw data and the model’s reasoning, allowing it to link data characteristics to algorithm suitability.

Preference Corpus

Execution traces were collected from two real ML agents (AIDE and AutoMind) on MLE‑Bench, covering computer vision, NLP, and data‑science tasks (26 tasks total). After deduplication, taxonomy labeling, and expert sampling, the authors obtained 895 high‑quality instances, forming 18,438 pairwise comparisons.

Main Results

Using the Preference Corpus, DeepSeek‑V3.2 (Thinking mode) achieved 61.5% average accuracy, GPT‑5.1 reached 58.8%, both significantly above the random baseline (50.0%) and the complexity‑heuristic baseline (50.8%). The improvement stems from the model extracting meaningful signals from the semantic data report rather than relying on code length or model size.

Analysis of Factors

Providing the verified data report yields the largest gain; mismatched data context offers almost no improvement.

Reasoning modes such as Thinking/CoT outperform direct answering because they force the model to first understand data, then analyze code, and finally compare.

Model size shows diminishing returns after ~30B parameters, indicating that prediction ability relies more on semantic understanding than sheer scale.

Limitations

The approach works well for pairwise comparison (61.5% accuracy) but degrades to 31.1% for listwise ranking. The corpus, while large, is imbalanced across task types, and the verified reports for CV/NLP rely only on metadata, lacking deeper multimodal analysis. Moreover, the validation‑test gap (72.2%) shows that even real execution is not a perfect oracle.

FOREAGENT

Building on the findings, the paper proposes FOREAGENT, which replaces the traditional Execute‑then‑Learn loop with a Predict‑then‑Verify pipeline: generate many candidates in parallel, use the LLM‑based predictor to rank them, gate by a confidence threshold (c=0.7), and only execute the top‑k (k=1) candidates. On five AI‑for‑Science tasks from MLE‑Bench, FOREAGENT achieves an average 6× speedup, 3.2× more node exploration, and a 6% increase in Beat Ratio.

Significance

By moving part of the feedback mechanism inside the model, FOREAGENT reduces the need for costly execution, enabling more efficient AutoML and AI‑for‑Science workflows. The authors note that 61.5% accuracy is a lower bound and that future work should address listwise ranking, richer multimodal reports, and more aggressive inference‑time search strategies.

Paper: https://huggingface.co/papers/2601.05930

Code and data: https://github.com/zjunlp/predict-before-execute

Dataset: https://huggingface.co/datasets/zjunlp/PredictBeforeExecute