Can S‑Curve Theory Explain the Limits of Large‑Model Scaling Laws?

AI Intelligence S‑Curve

Empirical observations of AI progress suggest that intelligence growth follows a sigmoid (S‑shaped) curve rather than an unbounded exponential trend. Early research shows slow improvement, a rapid acceleration once key breakthroughs appear, and finally a plateau as the technology approaches its intrinsic limits. The sigmoid can be expressed as f(x)=\frac{1}{1+e^{-K(x-x_0)}}, where K controls the steepness and x denotes compute or data scale.

No infinite growth

Scaling‑law formulas that predict continual performance gains with more compute (e.g., Chinchilla or Gopher laws) are valid only within a limited window. Extending the horizon reveals a slowdown, implying an eventual ceiling for any single scaling law.

Superposition of S‑curves

The sum of several sigmoid functions is itself a sigmoid with a broader output range. If multiple abilities each follow their own sigmoid, the aggregate performance curve remains S‑shaped, merely scaled vertically.

Scaling Laws Across Model Phases

Three phases

Large language models exhibit three distinct phases, each with its own scaling behavior:

Pre‑training : training on massive text, code, and other data.

Reinforcement‑learning fine‑tuning (RL) : alignment or instruction‑following stage.

Inference / Test‑time : deployment‑time performance.

For each phase the performance P as a function of compute C can be approximated by a sigmoid:

P(C)=\frac{P_{max}}{1+e^{-K(C-C_0)}}

Pre‑training scaling law via ability decomposition

Model intelligence can be decomposed into three coarse abilities:

Language ability – high data density, steep sigmoid (large K).

World‑knowledge ability – moderate density, moderate K.

Logical‑reasoning ability – low density, shallow sigmoid (small K).

The overall pre‑training performance is the sum of the three sigmoids, preserving the S‑shape.

Ability‑density hypothesis

Define ability density

D_A = \frac{\text{tokens containing signals for ability }A}{\text{total training tokens}}

. A larger D_A yields a larger steepness parameter K_A, producing a faster learning curve. Language data have the highest density, while code, mathematics, and scientific problem data have low density, explaining the slow improvement of reasoning even with massive scale.

Practical ways to raise reasoning performance

Increase the proportion of high‑density reasoning data (code, math, scientific problems) in the training corpus, thereby raising D_{reasoning} and the corresponding K.

Place a large share of this reasoning‑rich data in the later (annealing) stage of pre‑training, which temporarily amplifies K for reasoning and yields a sharp gain in the final performance curve.

Future outlook for RL and test‑time scaling

Both RL fine‑tuning and inference phases are expected to follow their own sigmoids. When each reaches its plateau, a new scaling regime (a new S‑curve) may emerge through architectural or algorithmic innovations, creating the illusion of continued exponential growth—analogous to a “Moore’s law for large models”.

Illustrative Figures