Can a $10 Million Inference Budget Uncover AI’s Real Upper Limit?

As large language models (LLMs) tackle complex reasoning, automation research, and cybersecurity tasks, traditional benchmark tables that compress abilities into a single score are becoming insufficient.

OpenAI researcher Noam Brown notes that when models can use more reasoning steps, tools, or longer search times, a single number fails to reflect actual capability. He emphasizes that model performance now depends not only on the model itself but also on the amount of computation allocated during inference.

Brown illustrates this with the release of GPT‑5.5. Initial benchmark scores showed only modest improvement over GPT‑5.4, leading some users to doubt the new version. However, within hours of open access, developers testing more demanding tasks observed markedly better long‑chain reasoning and sustained execution, indicating that traditional scores missed a substantial capability gain.

The core issue is that different models are often evaluated under unequal inference budgets. Researchers typically tune test configurations to maximize scores, which can hide a model’s ability to improve when granted additional tokens, calls, or runtime. In a cybersecurity benchmark, GPT‑5.5’s advantage over GPT‑5.4 vanished under a fixed “maximum test compute” condition, yet became evident when token count, cost, or latency were held constant across models.

Brown proposes shifting from a single‑score view to a "performance‑inference‑budget" curve, plotting task performance (y‑axis) against computation resources such as token count, cost, or wall‑clock time (x‑axis). This approach answers questions like: which model performs better at the same budget, how quickly performance scales with budget, and whether a model is nearing its capability ceiling.

He acknowledges trade‑offs of each metric: token counts vary with tokenizers and generation speed; cost depends on hardware utilization and batching; runtime can be obscured by parallel generation techniques. Nonetheless, any of these variables convey more information than a budget‑agnostic score.

The discussion extends to AI safety. If a model’s abilities keep rising with more inference budget, safety assessments must consider the highest plausible budget an adversary might afford. Brown cites the Gemini 3 Deep Think controversy, where a high‑budget benchmark showed strong results but the accompanying safety report was missing, highlighting the need for systematic evaluation under varied budgets.

He suggests three concrete actions: (1) publish benchmark results across multiple inference budgets when releasing new models; (2) require benchmark leaderboards to record or standardize inference resource usage; and (3) incorporate inference budget considerations into AI preparedness frameworks and responsible scaling policies, including uncertainty estimates for high‑budget extrapolations.

While exhaustive high‑budget testing may be costly, Brown recommends testing within feasible budgets and extrapolating trends, clearly marking prediction intervals and uncertainties. This helps regulators and developers understand how risk boundaries might shift when models receive substantially more compute.

Overall, Brown predicts that inference budget will become a core parameter—alongside model size, data, and context window—in future AI capability assessments, moving the industry away from single‑number model rankings toward richer, budget‑aware evaluations.