Jun 27, 2026 · Artificial Intelligence

Lilian Weng’s Deep Dive Overturns Three Years of Large‑Model Scaling Law Assumptions

In a ten‑thousand‑word analysis, former OpenAI safety VP Lilian Weng retraces the history of model scaling laws from Kaplan’s 2020 formulation, demonstrates how DeepMind’s Chinchilla overturns the original parameter‑to‑data ratio, uncovers two critical bugs in the Chinchilla paper, and warns that the impending 2026‑2028 data wall makes naïve scaling of parameters and compute unsustainable.

AI trainingchinchilladata wall

0 likes · 10 min read

Lilian Weng’s Deep Dive Overturns Three Years of Large‑Model Scaling Law Assumptions

Fighter's World

Dec 21, 2024 · Artificial Intelligence

Is Pre‑training Coming to an End? Evaluating Data Sufficiency

The article examines Ilya Sutskever’s claim that pre‑training will end, argues that scaling laws still hold and data is not yet a bottleneck, highlights the scarcity of high‑quality frontier data, and explains why the industry is shifting toward inference‑time compute (o1) as a more sustainable path for large language models.

AI trendsInference‑time ComputePre‑training

0 likes · 13 min read

Is Pre‑training Coming to an End? Evaluating Data Sufficiency