Why Small LLMs Are the Secret Weapon for Scaling Large Model Research

Why Small LLMs Matter

Qwen2 is popular not because its technical report is exhaustive, but because it provides a complete "full‑stack" ecosystem that enables researchers to work with much smaller models sharing the same tokenizer and pre‑training data. These "homologous small models" often yield far more research value than the 72B Qwen2 itself.

Key Concepts

Homologous small model : a smaller LLM trained with the same tokenizer and the same 7‑trillion‑token pre‑training dataset.

Small model : any compact model (e.g., BERT, RoBERTa, XGBoost, logistic regression) whose primary advantage is speed or low resource consumption, regardless of training details.

Homologous Small Models as an Experimental Playground

Scaling laws show that the performance of small models can predict the performance of larger ones. Consequently, many pre‑training and post‑pre‑training questions—optimal data mix, curriculum order, data quality thresholds, token‑length strategies, learning‑rate schedules—can be answered cheaply by training a small model for a few days (e.g., 100 B tokens per day) and observing loss curves, benchmarks, or SFT results.

During the alignment stage, small models can still be useful for estimating the impact of large data collections on a specific capability, but their guidance is weaker because alignment data are usually small enough to train the target large model directly.

Small models require larger learning rates; large models need smaller rates due to sparse feature spaces.

Mixture‑of‑Experts (MoE) scaling laws are still immature, and MoE models often lack a stable training architecture comparable to LLaMA.

Large Models as Teachers

Large models set the performance ceiling that small models cannot surpass, but they also provide valuable supervision through techniques such as model distillation and reward modeling.

Model Distillation

Distillation transfers knowledge from a large model to a small one, ideally using the same tokenizer. Traditional distillation uses soft labels (probability distributions over the vocabulary) rather than hard one‑hot labels, which conveys more information. For example, Google’s Gemma small model benefits from this approach.

Because distillation requires storing logits for the entire vocabulary, a practical compromise is to clip and keep only the top‑N logits.

Reward Modeling

Using a large model as a reward model for a small model is increasingly popular. When the reward model shares the same pre‑training data as the policy model, both have comparable knowledge, leading to fairer scoring and reduced hallucinations during alignment.

The Many Small Models Behind a Large Model

Large models rely on numerous auxiliary small models:

Data‑quality classifier : often a RoBERTa‑style model that scores pre‑training data, as seen in LLaMA 3 and Qwen2 pipelines.

Domain classifier : extracts high‑quality domain‑specific data for post‑pre‑training.

Online model classifier : decides whether to invoke RAG, safety checks, or tool usage at inference time.

RAG model : retrieves relevant documents; BGE is a common choice.

Data generation models : small models fine‑tuned on high‑quality data can replace expensive GPT‑4 calls, e.g., using a 0.5 B model to produce task‑specific data via SFT.

Practical tricks include repurposing a small model as a binary classifier (outputting only 0 or 1) and swapping the language‑model head with a reward‑model head for more efficient scoring.

Empirical observations show that larger models, while having higher ceilings, are more prone to over‑fitting if the training data are biased or imbalanced. A 0.5 B model may outperform a 1.5 B model when the former is trained on a more diverse data mix.

Conclusion

Although large LLMs have solved many long‑standing NLP challenges, small models remain indispensable for rapid experimentation, data quality filtering, domain adaptation, and cost‑effective data generation. Researchers should consider whether a small model can address a problem before defaulting to expensive GPT‑4 prompts.

Zhihu: https://zhuanlan.zhihu.com/p/714399961