Lao Guo's Learning Space

Aug 20, 2026 · Artificial Intelligence

How Transformers Use Positional Encoding to Distinguish "I love you" vs "You love me"

The article explains why vanilla Transformers cannot sense word order, introduces sinusoidal positional encoding as the elegant solution, compares it with learned and relative encodings, and shows how modern variants like RoPE preserve both absolute and relative positions for long texts.