How Transformers Use Positional Encoding to Distinguish "I love you" vs "You love me"

1. Transformers lack inherent order awareness

Self‑attention processes all tokens simultaneously and is permutation‑invariant, so without additional information it cannot distinguish sequences such as “I love you” and “You love me”.

2. Why naïve integer position IDs fail

Range mismatch : token embeddings are typically bounded (≈‑1 to 1) while integer positions grow without bound, causing positional information to dominate semantic content.

No relative distance : absolute IDs cannot express that positions 1‑3 and 101‑103 are two steps apart.

Poor extrapolation : models trained on ≤ 100 positions cannot handle unseen positions (e.g., 150) at inference.

The 2017 “Attention Is All You Need” paper introduced sinusoidal positional encoding to address these issues.

3. Sinusoidal encoding formula

Even dimensions: PE(pos, 2i) = sin(pos / 10000^{2i/d}) Odd dimensions: PE(pos, 2i+1) = cos(pos / 10000^{2i/d}) Each dimension behaves like a clock hand with a distinct frequency; the combination of sine‑cosine values uniquely identifies a position. Sin and cos appear together because they are 90° out of phase, providing a complete description of a single frequency.

4. Free relative positioning

For any fixed offset k, PE(pos + k) is a linear transformation of PE(pos). Consequently the model only needs to learn a single “move k steps” rule to capture all relative distances. The periodic nature of sine and cosine also enables graceful extrapolation to positions far beyond the training range.

5. Positional‑encoding workflow

Input : convert each token to an embedding (what the word is).

Add position : add the same‑dimensional positional vector element‑wise (where the word is).

Self‑attention : feed the combined vectors to attention layers, allowing the model to attend based on both content and order.

6. Subsequent alternatives

Learned positional encoding (BERT, GPT): each position has a trainable vector. Flexible and often yields higher accuracy, but cannot extrapolate beyond the maximum trained length.

Relative positional encoding (T5): encodes the distance between token pairs directly, improving relational modeling at the cost of higher computational overhead.

Rotary Positional Encoding (RoPE) (LLaMA, Qwen): rotates token embeddings instead of adding a separate vector, preserving sinusoidal relative benefits while supporting very long contexts (e.g., 128 K tokens). Reported to enable large models to handle up to 128 K or more context length.

7. Resolving the “I love you” vs “you love me” example

Because the word “I” occupies different positions, its positional vector differs in the two sentences. The added positional encoding makes the self‑attention layers aware of order, allowing the model to distinguish “I before you” from “you before I”.

Thus a lightweight positional‑encoding component converts a permutation‑invariant self‑attention mechanism into an order‑sensitive model, forming a foundational element of modern large‑scale language models.