Did OpenAI’s Original Scaling Law Contain a Fatal Bug That Wasted Trillion‑Scale Compute?

For the past five years the AI field has been driven by a scaling law that suggested, under a fixed compute budget, model size should grow faster than data, with optimal parameters proportional to compute^0.73. OpenAI’s 2020 conclusion that larger models like GPT‑3 (175 B parameters) would yield the best performance cemented this belief.

Diogo Almeida, a former OpenAI optimizer researcher, published a blog titled Scaling Laws, Honestly claiming the original scaling law was wrong because it contained a bug. The bug stemmed from setting the loss scale too high in the optimizer and averaging the Huber loss per sample instead of summing it, causing the fitting process to terminate prematurely.

This error caused the industry to allocate vast compute resources to overly large models while feeding them insufficient data, effectively burning a trillion‑scale amount of GPU time on ineffective scaling.

DeepMind’s Chinchilla paper later challenged the original law, showing that model size and data should be scaled together—approximately 20 tokens per parameter—for optimal efficiency. However, Besiroglu et al. (2024) re‑examined Chinchilla’s data points and discovered another bug: the loss‑scale issue persisted, and the cosine learning‑rate decay forced the learning rate to zero near the training endpoint, creating an artificial performance plateau.

The article outlines a three‑step “deception”: (1) Data confinement —using the same token budget for all models, starving large models; (2) Learning‑rate decay —cosine decay masks continued learning potential; (3) Authority arrogance —claims that results are largely independent of the learning‑rate schedule, hiding the underlying bug.

Researcher Adam Zachary Wasserman further argues that even a corrected scaling law remains English‑centric. In an experiment with identical architectures and compute, French models achieved comparable grammatical ability 50–100× more efficiently than English models, because English’s morphological sparsity forces models to rely on massive data.

Consequently, current compute‑allocation strategies are biased toward an inefficient language, wasting countless H100 GPU hours, electricity, and delaying the arrival of a more efficient AI era.