Inside Llama 3: Training, Architecture, and Performance Secrets

Introduction

This article provides a comprehensive, yet concise, overview of the Llama 3 family of foundation models, covering pre‑training, post‑training, inference optimizations, and benchmark performance.

1. Modern Foundation Model Training

1.1 Training Stages

Pre‑training : Large‑scale next‑token prediction on massive web corpora.

Post‑training : Instruction‑following (SFT), reward‑model alignment (RM), and Direct Preference Optimization (DPO) among other techniques.

1.2 Key Levers

Meta identifies three critical factors: data, scale, and managing complexity.

Data : Llama 3 uses ~15 T multilingual tokens, a significant increase over Llama 2’s 1.8 T.

Scale : Model sizes 8 B, 70 B, and 405 B are evaluated on a range of benchmarks.

Complexity Management : The architecture remains a dense Transformer without Mixture‑of‑Experts, simplifying training and inference.

1.3 Benchmark Highlights

Performance on MMLU, IFEval, code, math, reasoning, tool use, long‑context, and multilingual tasks is reported. The 70 B model approaches Claude 3.5‑sonnet on IFEval, while the 405 B model offers marginal gains over 70 B for many tasks but incurs higher compute cost.

2. Pre‑Training

2.1 Data Curation

Web Data Curation : De‑duplication at URL, document, and line levels; heuristic filtering (n‑gram repeats, toxic content); KL‑divergence based outlier removal.

PII & Safety Filtering : Removal of personally identifiable information and adult content.

Text Extraction & Cleaning : HTML parsing, removal of Markdown markers.

Model‑Based Quality Filtering : FastText and RoBERTa classifiers to label high‑quality samples.

The final mix is roughly 50 % general knowledge, 25 % math/reasoning, 17 % code, and 8 % multilingual data.

2.2 Data Mix & Annealing

High‑quality code and math data are used for a short learning‑rate annealing phase (last 40 M tokens) to boost downstream benchmark scores, especially for smaller models.

2.3 Model Architecture

Llama 3 retains a dense Transformer backbone but introduces several efficiency improvements:

KV Cache : Caches intermediate key/value states to avoid recomputation during autoregressive decoding.

Grouped Query Attention (GQA) : Reduces KV cache size by sharing keys/values across query groups, essential for models ≥70 B.

RoPE (Rotary Positional Embedding) : Replaces sinusoidal encoding to improve relative position handling.

2.4 Scaling Laws

OpenAI‑style scaling laws are revisited: compute (FLOPs), model parameters, and data tokens follow a power‑law relationship. Llama 3 also predicts downstream task loss directly (two‑stage scaling) to better correlate with actual benchmark accuracy.

2.5 Training Recipe

Initial Pre‑training : AdamW, peak LR ~?, linear warm‑up 8 k steps, cosine decay over 1.2 M steps; batch size grows from 4 M to 16 M tokens.

Long‑Context Pre‑training : Gradual increase of context window from 8 K to 128 K tokens, consuming ~0.8 T tokens.

Annealing : Final LR linearly decays to zero while focusing on high‑quality code and math data.

3. Post‑Training

3.1 Reward Model (RM)

Human preference data (A ≫ B ≫ C) is used to train a scalar reward model that scores generated text for safety, helpfulness, and alignment.

3.2 Supervised Fine‑Tuning (SFT)

SFT uses cross‑entropy loss on curated instruction‑following data, including rejection‑sampling outputs, synthetic task data, and a small amount of human‑annotated examples.

3.3 Rejection Sampling

Multiple responses are sampled for a prompt; the RM scores them and the highest‑scoring response becomes SFT data. PagedAttention is employed to share KV cache across sampled responses, reducing memory overhead.

# DPO data format
{
    'prompt': '',
    'chosen': '',
    'rejected': ''
}

3.4 Direct Preference Optimization (DPO)

DPO merges SFT and RM objectives, eliminating the separate RM training step. It masks formatting tokens to avoid conflicting gradients and adds an NLL loss on the chosen response to preserve its probability.

3.5 Data Processing & Quality Control

Topic Classification : Small models classify data into coarse/fine categories (e.g., math‑reasoning).

Quality Scoring : RM and Llama‑based classifiers assign scores; top‑quartile samples are kept.

Difficulty Scoring : Instag and Llama prompts estimate sample hardness for curriculum learning.

Semantic Deduplication : RoBERTa clustering followed by cosine similarity pruning.

4. Inference Optimizations

4.1 Parallelism

Both data parallelism (replicating the model across devices) and model parallelism (tensor + pipeline) are used. Llama 3 405 B runs on two nodes (16 GPUs total) with tensor parallelism inside a node and pipeline parallelism across nodes (Gpipe).

4.2 Quantization

FP8 inference leverages H100 native support, quantizing most feed‑forward matrices while keeping self‑attention in higher precision. Experiments show up to 50 % throughput increase with negligible loss in reward‑model scores.

5. Conclusions

Llama 3 demonstrates that scaling data quality, employing efficient attention mechanisms (GQA, KV cache, PagedAttention), and iterating post‑training with reward‑guided DPO can yield strong performance across diverse benchmarks while keeping inference costs manageable through parallelism and FP8 quantization.