Skip to content
Tagged articles

RMSNorm

4 articles · Page 1 of 1
ThinkingAgent
ThinkingAgent
Aug 4, 2026 · Artificial Intelligence

How Transformers Compute Contextual Relationships

The article explains how the Transformer architecture replaces RNNs with self‑attention, detailing the Q‑K‑V mechanism, positional encodings such as RoPE, multi‑head attention, modern improvements like SwiGLU and RMSNorm, and provides formulas for parameter and FLOP estimation.

FFNPositional EncodingRMSNorm
0 likes · 26 min read
How Transformers Compute Contextual Relationships
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Mar 20, 2026 · Artificial Intelligence

Why Kimi Dropped Residual Connections: A First‑Person Deep Dive into Attention Residuals

This article explains how Attention Residuals (AttnRes) replace traditional residual shortcuts with layer‑wise attention, details the mathematical reformulation, design constraints, static‑Q trick, full and block variants, and presents experimental evidence of significant accuracy gains with modest overhead.

AttentionNLPRMSNorm
0 likes · 11 min read
Why Kimi Dropped Residual Connections: A First‑Person Deep Dive into Attention Residuals
Code Archaeology
Code Archaeology
Oct 20, 2025 · Artificial Intelligence

LLaMA 2 & 3 Deep Dive: Architecture Innovations (RMSNorm, RoPE, GQA) and Post-Training Pipeline

This article explains Meta's LLaMA model series, detailing LLaMA 2's architectural innovations—RMSNorm, SwiGLU, rotary positional encoding, and grouped-query attention—and LLaMA 3's post-training pipeline including supervised fine-tuning, reward modeling, rejection sampling, and direct preference optimization.

DPOGrouped-Query AttentionLLM Architecture
0 likes · 10 min read
LLaMA 2 & 3 Deep Dive: Architecture Innovations (RMSNorm, RoPE, GQA) and Post-Training Pipeline