Skip to content
Tagged articles

SwiGLU

2 articles · Page 1 of 1
Code Archaeology
Code Archaeology
Oct 20, 2025 · Artificial Intelligence

LLaMA 2 & 3 Deep Dive: Architecture Innovations (RMSNorm, RoPE, GQA) and Post-Training Pipeline

This article explains Meta's LLaMA model series, detailing LLaMA 2's architectural innovations—RMSNorm, SwiGLU, rotary positional encoding, and grouped-query attention—and LLaMA 3's post-training pipeline including supervised fine-tuning, reward modeling, rejection sampling, and direct preference optimization.

DPOGrouped-Query AttentionLLM Architecture
0 likes · 10 min read
LLaMA 2 & 3 Deep Dive: Architecture Innovations (RMSNorm, RoPE, GQA) and Post-Training Pipeline
AIWalker
AIWalker
Mar 4, 2024 · Artificial Intelligence

How LLaMA Invades Computer Vision: Introducing VisionLLaMA from Meituan and Zhejiang University

VisionLLaMA extends the LLaMA transformer architecture to 2‑D visual data with naive and pyramid designs, introduces a 2‑D rotary position encoding (AS2DRoPE), and achieves faster convergence and superior performance over state‑of‑the‑art ViT models on image generation, classification, segmentation, and detection tasks.

AS2DRoPEComputer VisionLlama
0 likes · 6 min read
How LLaMA Invades Computer Vision: Introducing VisionLLaMA from Meituan and Zhejiang University