Why Small Models Can Never Match Large Models, Even with Unlimited Data

The article analyzes scaling laws and synthetic experiments to show that, due to power‑law data distributions and interference, some tasks remain unreachable for small models even with infinite data, a finding confirmed on real LLMs such as OLMo.

Scaling Lawsinterferencelarge language models

0 likes · 10 min read

Why Small Models Can Never Match Large Models, Even with Unlimited Data