Information Access vs. Reasoning: Experimental Attribution Analysis of LLM Agent Performance

ContextEval

ContextEval is a controlled evaluation framework that varies only the information visible to an LLM agent while keeping the prompt and model (GPT‑4o‑mini) fixed, allowing measurement of how context exposure alone influences optimization behavior.

Test Method

The experiments manipulate four orthogonal context dimensions: task description (full Kaggle spec), metric exposure (evaluation rules), parameter bounds (explicit search space), and feedback depth (history length of 1 or 5 steps). This yields a full‑factorial grid of 16 context strategies, each evaluated on four HPO benchmarks.

Initial configurations are generated by Sobol sampling (256 samples) and three stratified starting points are selected: low‑quality (bottom 20 %), average (mid‑range), and high‑quality (top 20 %). Performance is measured by Normalized Regret, the standardized distance to the optimal configuration.

Results

Initialization dominates. The strongest predictor of success is the starting configuration, not the agent’s actions. Agents beginning from poor points improve quickly but plateau; those starting near optimal improve little or even regress on the NOMAD benchmark.

Feedback depth paradox. Providing longer history (fd=5) consistently worsens Normalized Regret across all benchmarks, especially on Jigsaw, because low‑score records anchor the agent and restrict exploration.

Feasibility vs. optimization quality. Enforcing parameter bounds eliminates 96–100 % of invalid proposals, yet final performance does not improve, indicating that obeying constraints is not equivalent to achieving better optimization.

Random search comparison. LLM‑guided optimization is not reliably superior to random search; on the complex Jigsaw benchmark, a blind random algorithm outperforms the LLM even with full context and history.

Task context impact. Supplying the full task description yields limited benefit and can increase instability; performance appears driven more by pretrained priors than by iterative reasoning.

Agent can fix terrible configurations but struggles to make meaningful improvements on already good ones.

Are Agents Smarter or Just Better Informed?

LLMs heavily rely on contextual cues to activate pretrained priors. When given task descriptions or metric signals, they infer plausible hyper‑parameter ranges from training data rather than performing genuine reasoning based on observed feedback.

In practice, agents behave like feedback‑driven heuristics rather than true search algorithms, often failing to surpass random exploration on difficult tasks.

Significance of the Framework

By treating information exposure as a controlled variable, ContextEval isolates whether performance gains arise from reasoning or from richer metadata, informing better hot‑start strategies and more reliable agent benchmarking.

Future benchmarks should report context visibility; without it, LLM agent capabilities are easily over‑estimated.

Implications for AI Evaluation

Benchmarks that omit context visibility provide incomplete pictures: an agent that excels under full context may not be intrinsically smarter—it may simply have access to more information.