AI’s Shift from Gold Medals to Cost‑Effective Quantitative Success

AI breakthroughs in mathematical problem solving

Google’s Gemini‑Advanced model solved five of the six ultra‑hard problems in the most recent International Mathematical Olympiad (IMO), achieving a gold‑medal level (35/42). This marks the first time an AI system has been officially recognised by the IMO committee as a gold‑medal performer.

Expert perspective on evaluation

Fields Medalist Terence Tao, who attended the IMO award ceremony, praised the achievement but warned that without a controlled, standardised testing framework comparisons between AI models can become overly simplistic and misleading.

From qualitative to quantitative assessment

As AI technologies mature, the focus shifts from “who was first” to measurable metrics such as:

Compute resources (GPU hours, electricity consumption)

Required expertise (human supervision, domain knowledge)

Environmental impact (carbon footprint)

Safety and reliability risks

Cost example: An advanced AI tool that costs $1,000 in compute per attempt and succeeds on 20 % of attempts has an effective cost of $5,000 per successful solution. Reporting only the successful cases would severely under‑state the true expense.

In addition, the “stand‑by” cost of highly paid experts who monitor the system and are ready to intervene—even if no intervention occurs—must be accounted for in the total cost of deployment.

Scaling laws and tool categories

Scaling laws suggest that the most resource‑intensive AI systems tend to be the most powerful, yet both lightweight and heavyweight tools have distinct roles:

Lightweight tools – inexpensive, fast, and suitable for the majority of routine tasks.

Heavyweight tools – large‑scale models or specialised automated theorem provers (ATPs) that handle the hardest problems.

In Tao’s recent “Equational Theories Project”, 22 million implication proofs were required. The distribution of effort was:

~90 % solved by very cheap, brute‑force methods.

~9 % solved by medium‑strength ATPs.

~0.9 % solved with human expert assistance.

~0.1 % required collaboration between humans and high‑cost AI systems.

Although large language models were not heavily used, the project illustrates a typical progression: inexpensive AI handles the bulk of work, while expensive AI is reserved for the most challenging cases and is often combined with expert insight.

Recommendations for future benchmarking

To obtain reliable, comparable measurements of AI progress, future benchmarks and competitions should require:

Pre‑disclosure of total compute resources (GPU hours, energy consumption).

Detailed reporting of both successful and failed attempts to calculate true success rates.

Documentation of any human supervision or “stand‑by” costs.

Standardised test suites that are reproducible across research groups.

Such transparent reporting will enable the community to assess AI systems on cost‑effectiveness, safety, and scalability rather than solely on headline achievements.

Illustrative figures

Code example

来源：机器之心
本文
约2000字
，建议阅读
5
分钟
人工智能技术现已迅速接近从定性到定量成果的转型阶段。