Can AI Autonomously Plan Experiments to Discover Scaling Laws? SLDBench v2 Reveals the Gap

SLDBench v2: Active Scaling Law Discovery Benchmark

SLDBench v2 introduces an Active setting where AI must autonomously choose which experiments to run from a priced catalog of real training records, paying the actual compute cost for each, then predict unseen larger-scale experiments. The benchmark is built from 127 scenarios and 3.37 million training records, filtered to 30 tasks where even the classic power-law scaling law fails to fit. Candidate experiments span a median price ratio of 9,510×, and target test experiments are typically 7.4× more expensive than the candidates.

Experimental Setup

The benchmark uses historical experiment replay : publicly released training records are organized into an "experiment store" where each experiment shows its configuration, group, and price (actual GPU-hours consumed). Results are hidden until purchase. No budget cap is imposed, but every expenditure is logged. The 30 tasks come from 14 data sources, totaling 339k observations.

Eight Frontier AI Systems Evaluated

Systems cover three frameworks: Claude Code, Codex, and Gemini CLI. Prediction quality measured by R² (1 = perfect, 0 = predicting mean).

Passive Setting (all results given upfront)

All eight systems achieve nearly identical R² scores, median range 0.723–0.753.

Active Setting (AI chooses experiments)

Every system's performance drops; spread widens to 0.33 R² points.

Claude Opus 5.5 ranks first with median R² 0.570 while spending only 4.01% of full compute budget — beating the classic power-law formula fitted on all data (R² 0.455).

Top four systems surpass the power-law baseline using just 4–35% of the cost.

Buying all candidate experiments yields scores comparable to Passive, confirming the gap comes from which experiments are purchased, not from batch purchasing per se.

Purchase Behavior Analysis

All models prefer cheap experiments: each system's share of purchased points exceeds its share of cost. Purchases typically start cheap and shift toward expensive across batches. 16.7% of runs cover all groups but lose parameter variation; 10.7% miss groups entirely.

Failure Case: DeepSeek V4 Pro on Vision-Language Transfer Task

Task has 22 groups. After determining curve shape on a few cheap groups, the model bought only two early small-scale points in each of 14 other groups and assumed the same power exponent. This unverified assumption caused 96.4% of the test error. The error stems not from buying few points, but from never testing the critical hypothesis that different groups share the same curve shape.

Why Opus Succeeds: Three Distinct Behaviors

Buys fewest experiments : 21.5% of candidates (vs. 34–48% for other low-spend systems).

Concentrates expensive purchases late : First-batch price position 0.197, last-batch price position 0.608 (highest among systems). It first uses cheap runs to hypothesize a formula form, then targets costly experiments to test specific questions (e.g., "does the curve hold at larger scale?").

Most stable repeated runs : Lowest proportion of runs where R² swings between positive/negative or error varies >10× across three repetitions.

Gemstones Task Walk-through

On the Gemstones task (a suite of language models trained for scaling law research), Opus first buys cheap small-scale experiments varying model size, training tokens, and groups to form an initial formula. Each subsequent expensive purchase targets a concrete question: does the curve hold at larger scale? Does extended training saturate? In one run, Opus predicted a large-scale result, bought that experiment to verify, then revised the formula. Final cost: 0.09%–0.66% of full budget for R² 0.95–0.99.

Non-AI Control Experiment

Under identical formulas and budgets, a rule that selects points covering different price positions within each group outperforms buying only the cheapest points in 63%–77% of tasks. This confirms the principle: covering diverse price positions per group beats shallow coverage of many groups .

Conclusion & Future Direction

Current AI cannot yet autonomously plan experiments at the level required for large-scale model development. However, the best models already exhibit a scientist-like cycle: hypothesize → test → revise. Future work should focus on selecting experiments that discriminate between candidate scaling laws rather than merely minimizing cost.

References

Paper (arXiv:2610.16297): https://arxiv.org/abs/2610.16297

Project Page: https://lingbei-meng.github.io/SLDBench-v2/

Code & Data (GitHub): https://github.com/lingbei-meng/SLDBench-v2

Code example

来源：ScienceAI
本文
约4000字
，建议阅读
8
分钟
让AI把有限的预算花在能够区分假设对错的关键实验上。