Smol Training Playbook: Secrets to Building World-Class LLMs

HuggingFace’s SmolLM3 team released the technical document “The Smol Training Playbook: The Secrets to Building World-Class LLMs”. SmolLM3 is a 3 B‑parameter model—significantly larger than nanochat—and its performance surpasses Llama‑3.2‑3B and Qwen2.5‑3B.

The model uses a 36‑layer Transformer decoder, a document‑mask technique for long‑context training, NoPE, a 128 k context window, and Grouped‑Query Attention (GQA) borrowed from the Llama series. Training employs AdamW with a peak learning rate of 2e‑4, gradient clipping at 1.0, and weight decay of 0.1.

SmolLM3 introduces a dual‑mode inference architecture activated via system prompts /think and /no_think. In “think” mode the model generates reasoning traces (useful for math proofs or code debugging); in “no_think” mode it replies directly, achieving roughly a 50 % speed increase.

The core training philosophy is “broad then refined”. The authors describe a three‑stage pre‑training pipeline: start with large, generic data sources, then progressively add smaller, higher‑quality, domain‑specific data. They stress that data curation typically yields larger performance gains than architectural or optimizer tweaks.

To ensure results are attributable, the team conducts single‑variable ablations—changing only one data factor (e.g., mixing ratio) while keeping architecture and hyper‑parameters constant. When high‑quality data are introduced, they use annealed ablation and warn that over‑reliance on a tiny high‑quality set can cause duplication and hurt performance.

Preference data generation is performed by labeling outputs from a strong model (e.g., Qwen3‑32B) as “preferred” and from a weaker model (e.g., Qwen3‑0.6B) as “rejected”. Multiple candidates are generated per prompt, scored by an external reward model, and assigned preference tags.

For inference‑data generation, the team synthesises multi‑turn dialogues using a strong model to expand single‑turn instructions and produce reasoning traces (e.g., the IFThink dataset). Parallel data containing both /think and /no_think variants teach the model to switch behavior based on the prompt.

Post‑training, the “alphabet” consists of instruction‑tuning (SFT) as the foundation, followed by preference optimisation (DPO or APO) and, optionally, reinforcement learning (GRPO/RL). SmolLM3 adopts Anchored Preference Optimisation (APO). APO‑zero is used when the preferred responses are far superior to the current SFT checkpoint, encouraging improvement while preventing over‑optimisation; APO‑down is applied for safety‑oriented downgrades.

Model merging is described as a way to combine independently trained specialists (e.g., a code‑focused model and a reasoning‑focused model) by merging their weights, yielding a single model with combined capabilities without retraining.

The authors provide a decision‑flow checklist for starting a new LLM project: evaluate whether to train from scratch, consider existing models, prompt engineering, fine‑tuning needs, team experience, timeline, and target hardware (dense models for edge devices vs. MoE for cloud).

Training‑stability techniques include Z‑Loss regularisation, removing weight decay from embedding layers, and applying QK‑Norm (validated by OLMo2). Operational safeguards comprise a warm‑up phase (first 100 steps with 1 % data to verify NCCL), fault‑injection tests, throughput logging (tokens / s, MFU, GPU temperature), and automatic checkpoint rollback on NaN loss.

Context‑length scaling follows a staged approach: pre‑train at 4 k tokens, then extend to 32 k, then to 64 k, with inference‑time extrapolation to 128 k using YaRN.

Checkpointing is performed every 2‑4 hours; checkpoints are uploaded to cloud storage (e.g., S3) and the local copy deleted to free space. An anecdote warns that a stray rm -rf command deleted an entire checkpoint directory, costing a day of training.

Hardware‑utilisation metrics (GPU utilisation, MFU, temperature) are monitored, and scripts automatically restart training after failures.

In summary, building a world‑class LLM with a small team demands disciplined data quality work, systematic single‑variable experiments, robust infrastructure, and careful cost‑benefit analysis of every architectural or training change.