Master the Three Essential LLM Training Stages for 2025

DeepSeek‑R1 recently emerged, delivering a powerful inference model that rivals OpenAI’s o1 while costing far less, sparking a sharp drop in AI‑related stocks; the article uses this event to introduce the three‑stage training pipeline that underpins such models.

01 Pre‑training

Pre‑training is an unsupervised learning phase where the model ingests massive amounts of unlabelled text to acquire language patterns and a statistical world model. For example, given the prompt "Over there, I see", the model is likely to predict the next word "an". During this phase the loss function continuously adjusts parameters so the model outputs expected tokens, producing a "base model". Most base models are publicly available on HuggingFace, but they usually require further optimization before production use.

02 Supervised Fine‑Tuning (SFT)

Supervised fine‑tuning adopts a supervised learning approach, using labelled data to steer the model toward desired output behavior.

The base model often exhibits shortcomings; SFT aims to improve output quality and safety. It brings two main benefits:

Improved usefulness: a pre‑trained model may answer correctly but lack empathy; SFT can make responses more empathetic.

Enhanced compliance: the base model might generate harmful content, while SFT filters such outputs, refusing requests like "how to make dangerous items".

After SFT, the model is referred to as an Instruct model , capable of more natural and safe conversational behavior.

03 Reinforcement Learning from Human Feedback (RLHF)

RLHF uses human preference signals to create a reward model that guides reinforcement‑learning algorithms, encouraging the LLM to produce more helpful answers. In practice, users compare two model responses, select the better one, and this choice forms the reward signal that steers subsequent generation.

Summary

The three stages—pre‑training, supervised fine‑tuning, and RLHF—constitute the full training pipeline for modern large language models. Companies like DeepSeek are now achieving performance comparable to OpenAI’s flagship models and openly sharing their methods, opening new avenues for further innovation.