From Zero to LLM: The Five‑Stage Pipeline Behind GPT and Claude

Stage 1 – Data Collection and Tokenization

Before any model exists there is raw text. The pipeline starts by gathering massive amounts of public internet content, books, and code repositories. Because the raw material is noisy, most of the work is cleaning: removing garbage, deduplicating repeated paragraphs, and filtering harmful content. After cleaning the text is passed through a tokenizer that splits it into tokens (e.g., the word “tokenization” becomes three or four tokens). The output of this stage is a huge, clean, tokenized dataset; the model has not learned anything yet.

Experiment: run a small tokenizer on a sample text to see how it splits words.

Practice: clean a tiny corpus by deduplication and filtering low‑quality lines.

Compare: train a tiny model on clean vs. noisy data to observe the impact of data quality.

Stage 2 – Pre‑training (Next‑Token Prediction)

Pre‑training consumes the tokenized dataset and trains a model to predict the next token in a sequence. The objective is simple: given a series of tokens, guess the following one, compare the guess to the true token, and adjust billions of parameters accordingly. This process runs over trillions of tokens and yields a “base model” that has learned grammar, facts, reasoning patterns, and code syntax purely from the prediction task. The base model can generate fluent text but lacks any notion of helpfulness or safety.

Understand: explain the next‑token objective in one sentence.

Hands‑on: follow a beginner tutorial to train a mini language model on a small dataset.

Observe: notice how the model learns fluency but does not follow instructions.

Stage 3 – Supervised Fine‑Tuning (SFT)

To turn the base model into a useful assistant, supervised fine‑tuning supplies high‑quality instruction‑response pairs. Humans write or curate thousands of examples that demonstrate the desired behavior (question → helpful answer, command → appropriate response). The model is trained on this curated data with the same next‑token objective, but now the target outputs encode “usefulness”. The result is a model that follows instructions and produces helpful answers, though it may still lack safety and alignment.

Read example pairs from a base model and an SFT model to spot the difference.

Build a small instruction dataset and fine‑tune a mini open‑source model.

Notice the outsized effect of sample quality versus quantity.

Stage 4 – Reward Modeling

After SFT, the model can answer well but “good” answers are hard to define. The solution is to collect human preference data: for a single prompt, generate multiple answers, have humans rank them, and train a second model—the reward model—to predict those rankings. The reward model never talks to users; it acts as an automated judge that scores any answer according to human preference.

Grasp why ranking is more scalable than writing perfect answers.

See how the reward model learns to imitate human judgments.

Read about the data‑collection pipeline for preference comparisons.

Stage 5 – Reinforcement Learning from Human Feedback (RLHF)

The final stage combines the SFT model and the reward model in a reinforcement‑learning loop: the SFT model generates an answer, the reward model scores it, and the policy is updated to increase the score. Repeating this generate‑score‑improve cycle lets the model surpass the limits of the original human examples, producing more coherent, helpful, and safe responses. Variants such as RLAIF replace part of the human feedback with rule‑based or constitutional prompts.

Understand the loop: generate → score → improve.

Realise why this loop enables learning beyond the amount of human‑written data.

Read the distinction between RLHF and RLAIF.

By mastering this five‑stage pipeline—data, pre‑training, supervised fine‑tuning, reward modeling, and RLHF—you can construct a miniature LLM, diagnose why large models hallucinate, and appreciate how each stage contributes to the final helpful assistant.