LLaMA 2 & 3 Deep Dive: Architecture Innovations (RMSNorm, RoPE, GQA) and Post-Training Pipeline
This article explains Meta's LLaMA model series, detailing LLaMA 2's architectural innovations—RMSNorm, SwiGLU, rotary positional encoding, and grouped-query attention—and LLaMA 3's post-training pipeline including supervised fine-tuning, reward modeling, rejection sampling, and direct preference optimization.
