Skip to content
Tagged articles

Grouped-Query Attention

4 articles · Page 1 of 1
Code Archaeology
Code Archaeology
Oct 20, 2025 · Artificial Intelligence

LLaMA 2 & 3 Deep Dive: Architecture Innovations (RMSNorm, RoPE, GQA) and Post-Training Pipeline

This article explains Meta's LLaMA model series, detailing LLaMA 2's architectural innovations—RMSNorm, SwiGLU, rotary positional encoding, and grouped-query attention—and LLaMA 3's post-training pipeline including supervised fine-tuning, reward modeling, rejection sampling, and direct preference optimization.

DPOGrouped-Query AttentionLLM Architecture
0 likes · 10 min read
LLaMA 2 & 3 Deep Dive: Architecture Innovations (RMSNorm, RoPE, GQA) and Post-Training Pipeline
Tencent Cloud Developer
Tencent Cloud Developer
Feb 6, 2025 · Artificial Intelligence

DeepSeek V Series: Technical Overview of Scaling Laws, Grouped Query Attention, and Mixture‑of‑Experts

The article reviews DeepSeek’s V‑series papers, explaining how scaling‑law insights, Grouped Query Attention, a depth‑first design, loss‑free load balancing, multi‑token prediction and Multi‑Head Latent Attention together enable economical mixture‑of‑experts LLMs that rival closed‑source models while cutting compute and hardware costs.

DeepSeekGrouped-Query AttentionMixture of Experts
0 likes · 13 min read
DeepSeek V Series: Technical Overview of Scaling Laws, Grouped Query Attention, and Mixture‑of‑Experts
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Feb 5, 2025 · Artificial Intelligence

What Optimizations Power DeepSeek’s High‑Efficiency LLMs?

The article enumerates DeepSeek’s extensive technical optimizations—including Grouped Query Attention, Multi‑head Latent Attention, Mixture‑of‑Experts, 4D parallelism, quantization, and multi‑token prediction—that together enable cheap, high‑performance large language models.

4D parallelismDeepSeekGrouped-Query Attention
0 likes · 8 min read
What Optimizations Power DeepSeek’s High‑Efficiency LLMs?
Baobao Algorithm Notes
Baobao Algorithm Notes
Jul 31, 2024 · Artificial Intelligence

What Makes Mistral’s 7B, Mixtral, and Large 2 Models Stand Out? A Deep Technical Dive

This article compiles key technical details of the Mistral model family—including Mistral 7B, Mixtral 8×7B, Mixtral 8×22B, Mistral Nemo, and Mistral Large 2—covering their architectural innovations such as sliding‑window attention, grouped‑query attention, mixture‑of‑experts design, scaling parameters, performance benchmarks, quantization requirements, and practical deployment commands.

Grouped-Query AttentionMistralMixtral
0 likes · 17 min read
What Makes Mistral’s 7B, Mixtral, and Large 2 Models Stand Out? A Deep Technical Dive