Why Transformers Need Positional Embeddings and How They Work

Transformers excel at modeling relationships between any two tokens, but their self‑attention mechanism is blind to token order, treating a sentence as an unordered set of words. To give the model a sense of sequence, each token must receive a positional label.

A naïve approach of adding a large integer position index directly to the embedding vector overwhelms the semantic values (e.g., adding P(10000) = [10000, 0, 0] to E(apple) = [0.05, -0.02, 0.01] yields [10000.05, -0.02, 0.01]), causing the model to focus on position rather than meaning.

The proper solution is to generate position vectors whose magnitude is comparable to token embeddings while still encoding position information.

Classic Method 1: Sinusoidal (Fixed) Positional Encoding

Each position pos is mapped to a d -dimensional vector using sine and cosine functions of varying frequencies. The resulting values lie in [-1, 1], preventing the “scale‑overflow” problem, and the encoding can be computed for arbitrarily long sequences because it is a deterministic function.

Different dimensions use different frequencies, allowing distinction of both nearby and distant positions.

Example with a 3‑dimensional token vector: E(apple) = [0.05, -0.02, 0.01] combined with P_sin(10000) = [0.20, -0.91, 0.03] yields X = [0.25, -0.93, 0.04], preserving semantic scale.

Classic Method 2: Learnable Positional Embeddings

Models such as BERT allocate a fixed number of position slots (e.g., 512). Each slot starts with a random vector and is updated during training so that the sum E + P[pos] minimizes task loss. This produces position vectors tailored to the model, but the number of slots is fixed; positions beyond the allocated range have no embedding.

Example: after training, P_learn(3) = [0.12, -0.03, 0.07] added to E(apple) gives X = [0.17, -0.05, 0.08].

Modern Method: Rotary Positional Encoding (RoPE)

Instead of adding a vector, RoPE rotates each token vector in a plane by an angle that depends on its position. The rotation makes the attention computation depend only on the relative angle θ(pos2) – θ(pos1), effectively encoding relative positions.

Imagine a token vector as an arrow; at position 1 it is rotated 10°, at position 2 by 20°, etc. When two tokens interact, the model sees only the angle difference (e.g., 20°), capturing their relative distance.

Extending to Long Sequences: PI and YaRN

RoPE works well for lengths seen during training, but very long inputs can cause attention drift. Engineers apply two engineering tricks:

PI (Position Interpolation) : compress a long real position range (e.g., 0‑16000) into the trained range (e.g., 0‑4000) by scaling positions, effectively “zooming out” the sequence.

YaRN : a refined version of PI that scales dimensions differently—slow‑changing dimensions are compressed more heavily than fast‑changing ones—preserving global stability while keeping local detail.

For a model trained on up to 4 k tokens, PI would map position 12000 to 3000, whereas YaRN might apply a 4× slowdown to slow dimensions and only 1.5‑2× to fast dimensions, balancing coverage and precision.

Conclusion

We have covered the evolution of positional encoding in Transformers: sinusoidal fixed functions, learnable embeddings, rotary encoding, and the PI/YaRN extensions for long‑range handling. The next article will combine these insights with attention, normalization, and residual connections to dissect the full GPT architecture.