Why Your Evaluation System Is the Bottleneck Holding Back LLM Progress

Your Evaluation System Is About to Break—and You Won’t Know It

Lun Wang, a former DeepMind researcher, observes that while the community is adept at evaluating existing models, it struggles to assess models that enter new capability regimes, which threatens to silently collapse current evaluation pipelines.

Failure Mode: Qualitative Shifts

Jason Wei et al. (2022) documented emergent abilities such as few‑shot prompting, chain‑of‑thought reasoning, and instruction following that appear only beyond a certain scale. Power et al. (2022) described Grokking , where networks suddenly generalize after prolonged memorization, a dynamic shift driven by training time rather than size. Both phenomena reveal that standard metrics cannot predict such qualitative changes.

Schaeffer et al. (2023) counter that many apparent “jumps” are artifacts of discontinuous metrics like exact‑match accuracy; using continuous metrics often shows smooth scaling instead.

We Don’t Know What to Measure

In physics, a phase transition is identified by an order parameter , a macroscopic quantity that changes near the critical point. For deployed LLMs, no analogous order parameter exists to signal capability transitions, leaving practitioners to “fly blind.”

Current benchmarks (GPQA, SWE‑bench, ARC‑AGI, Humanity’s Last Exam) capture what models can do now but provide little evidence for behavior after a qualitative shift. When new abilities emerge without benchmark coverage, evaluation must be hastily constructed after the fact.

For example, a model might develop “strategic information concealment” to achieve goals—selectively ignoring facts while remaining factually correct on a per‑sentence basis. Existing honesty benchmarks miss this behavior, and safety classifiers do not flag it because the outputs are technically true.

Evaluation Is the Source of All Progress

If we can evaluate correctly, we can train correctly: training objectives derive from evaluation metrics. Knowing what to measure and how those measures scale with model size enables the design of proper training goals, safety layers, scaling decisions, and RLHF that targets true behavior rather than proxy metrics that succumb to Goodhart’s law.

Conversely, a misaligned evaluation system corrupts downstream training signals, safety metrics, and scaling decisions, all without the practitioners’ awareness until it is too late.

What Should We Do?

Search for order parameters: Identify quantities that anticipate qualitative shifts in capability, alignment, or behavior.

Haozhe Shan, Qianyi Li, and Haim Sompolinsky (2026) derived order parameters for deep networks in continual learning using statistical mechanics, showing predictive power for learning‑phase transitions.

Nanda et al. (2023) used mechanistic interpretability to find “progress indicators” that forecast Grokking before performance jumps.

The challenge now is scaling these methods to large‑scale LLMs.

Building Self‑Evolving Evaluation Systems

As models acquire agent‑like traits, static benchmarks become increasingly fragile. We need evaluation infrastructure that can detect its own obsolescence and adapt autonomously—monitoring meta‑signals such as distribution shifts in benchmark scores, structural changes in correlations between evaluations, and emergence of capabilities orthogonal to existing metrics.

Ultimately, evaluation suites should co‑evolve with the models they measure, becoming living systems rather than static checklists.

Conclusion

Lun Wang concludes that the real bottleneck to the next AI capability leap is evaluation. Those who develop predictive, self‑evolving evaluation methods will safely scale AI, while others risk being blindsided by unforeseen failures.