LLaMA 2 & 3 Deep Dive: Architecture Innovations (RMSNorm, RoPE, GQA) and Post-Training Pipeline

Meta's LLaMA series aims to compete with OpenAI through (nominally) open releases. So far four generations have been published: LLaMA 1 (Feb 2023, paper "LLaMA: Open and Efficient Foundation Language Models"), LLaMA 2 (Jul 2023, paper "Llama 2: Open Foundation and Fine-Tuned Chat Models"), LLaMA 3 (Apr 2024, paper "The Llama 3 Herd of Models"), and LLaMA 4 (Apr 2025, blog post "The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation"). LLaMA 4 adopts a mixture-of-experts architecture and will be covered separately; this article focuses on the first three models.

1. LLaMA 2 Model Architecture

LLaMA uses a Transformer decoder-only architecture with several key modifications, illustrated in a comparison diagram by Umar Jamil.

Normalization moved to the input position (pre-norm).

RMSNorm replaces LayerNorm.

FFN activation function changed to SwiGLU.

Positional encoding changed to Rotary Positional Encodings (RoPE).

Attention mechanism changed to Grouped-Query Attention (GQA).

1.1 RMSNorm

The purpose of RMSNorm is to reduce computation while achieving performance comparable to LayerNorm.

1.2 SwiGLU

When the parameter β=1, the activation function is known as SiLU. Its curve resembles ReLU but is smoother near zero. In practice, SiLU helps improve model accuracy .

1.3 Rotary Positional Encoding (RoPE)

RoPE is a key innovation in LLaMA: it transforms "position" from absolute indices into "relative phase", enabling the model to understand relative distance, which far surpasses traditional Transformer positional encoding in long-context and generalization capabilities .

In 2D space, rotation is intuitive. Define rotation matrix:

$$
\mathbf{R}_\theta =
\begin{bmatrix}
\cos\theta & -\sin\theta \\
\sin\theta & \cos\theta
\end{bmatrix}
$$

Multiplying a vector by this matrix rotates it by θ degrees in 2D.

Extension to high dimensions divides the space into multiple 2D subspaces, each with a different rotation speed.

1.4 Grouped-Query Attention (GQA)

The motivation for GQA is to improve inference efficiency and reduce inference cost .

Standard multi-head attention (left) generates equal numbers of K, V, Q per token. Multi-query attention (right) generates multiple Q but only one KV, maximally reducing parameter count and KV cache size . GQA (middle) is a compromise that maintains accuracy while reducing parameter count .

2. LLaMA 3 Post-Training

LLaMA calls the pre-trained model the base model . In LLaMA 2, the base model is fine-tuned to produce Llama 2 Chat; in LLaMA 3, the base model undergoes post-training to produce Llama 3 Instruct. Two terminology shifts occur:

Fine-tuning → post-training: LLaMA 3's term is clearer, distinguishing from traditional fine-tuning.

Chat → Instruct: Not a word game; Meta wants to convey that LLMs are not "chatbots" but "AI engines that understand instructions and execute tasks accurately" .

The post-training pipeline:

2.1 SFT (Supervised Fine-Tuning) — "Imitate Humans"

Purpose: Transform the model from "sentence completion" to "question answering".

Method:

Humans or models generate a batch of high-quality instruction data . Example:

User: Explain quantum computing.
Model: Quantum computing leverages superposition and entanglement for information processing…

Model performs supervised learning on this data: input question → output correct answer.

Result: The model learns to follow instructions, answer questions, and stay on topic. The output is the SFT Model (instruction-tuned model) .

2.2 Reward Model — "Learn to Score"

Purpose: Teach the model which answers align with human preferences.

Method:

For the same question, provide multiple candidate answers (human or system generated). Example:

Q: Explain black holes
A1: Black holes are extremely dense celestial objects…
A2: A black hole is a hole in space.

Humans (or model assistants) select the better answer.

Data becomes (Q, A1, A2, which is better).

Train a "scorer": input (Q, A) → output score (higher is better).

Effect: The reward model learns to judge answer rationality, logic, politeness, etc.

2.3 Rejection Sampling — "Filter Out Bad Answers"

Purpose: Select high-quality samples from model generations for the next training step.

Method:

Let the SFT model generate multiple candidate answers for the same question (e.g., 10).

Score them with the reward model.

Keep only the highest-scoring ones (e.g., top 20%).

Result: A "clean, high-quality" training set that aligns better with human preferences than the original SFT data.

2.4 DPO (Direct Preference Optimization) — "Directly Learn Preferences"

Purpose: Avoid complex reinforcement learning; directly teach the model what humans prefer.

Method:

Use reward model preference data (good answer vs. bad answer).

Directly optimize the model to output distributions closer to the good answer.

Advantages:

Simpler and more stable than RLHF.

No need for repeated sampling or reward tuning.

Performance matches or exceeds RLHF.

2.5 Final DPO Model — Llama 3 Instruct

The final model:

Understands complex instructions.

Answers align with human habits.

Polite, logical, less hallucination.

References

1.

https://arxiv.org/abs/2302.13971 2. https://arxiv.org/abs/2307.09288 3. https://arxiv.org/abs/2407.21783 4. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ 5. https://www.youtube.com/watch?v=Mn_9W1nCFLo