How Can Large Language Models Extend Their Context Window? A Deep Dive into Position Encoding

Absolute Positional Encoding

The original Transformer uses sinusoidal (triangular) absolute positional encoding. For token position k the encoding consists of two components per dimension pair p_{k,2i} and p_{k,2i+1} defined as

Each hidden‑state dimension is split into pairs; the pair forms a 2‑D vector that rotates with token index. Low‑dimensional pairs rotate quickly (high‑frequency) while high‑dimensional pairs rotate slowly (low‑frequency). The pair satisfies p_{k,2i}^2 + p_{k,2i+1}^2 = 1, giving a unit‑norm representation that is translation‑invariant. In the dot‑product attention, the query‑key term can be expressed using these sinusoidal vectors, leading to a closed‑form inner‑product that depends only on relative distance.

Relative Positional Encoding

Two major families are discussed:

ALiBi removes learned embeddings and adds a static bias that decays linearly with token distance.

Rotary Position Encoding (RoPE) treats each hidden‑state dimension pair as a 2‑D vector and rotates it by an angle proportional to the relative distance between tokens. The rotation matrix R(θ) satisfies R(θ)^T = R(-θ), guaranteeing that the relative position is preserved after rotation.

Mathematically RoPE can be written as a matrix multiplication:

Window Extrapolation Capability

LLMs are typically trained with a context window of 2k–8k tokens. When asked to process longer sequences, perplexity (PPL) spikes because low‑frequency components of the positional encoding are not well learned. Extrapolation aims to keep the model’s performance stable for unseen lengths.

Implementation Strategies

Four main families of extrapolation methods are commonly used:

Limited‑attention approaches : Reduce or eliminate attention to distant tokens (e.g., Sliding‑Window Attention, ALiBi) so the model never sees out‑of‑distribution positions.

Rotational‑speed adjustment : Map the relative rotation angles during inference back into the range observed during training. This includes simple linear position interpolation (Positional Interpolation) and NTK‑aware scaling (YaRN).

NTK‑by‑parts : Apply a ramp function that interpolates fully for low‑frequency dimensions and not at all for high‑frequency dimensions, preserving high‑frequency information while smoothing low‑frequency extrapolation.

Base‑size manipulation : Shrink or enlarge the base angle (the unit rotation) which changes how quickly the rotation cycles and thus how far the model can attend.

Typical formulas used in these methods are shown below:

Evaluation of Extrapolation

The “needle‑in‑a‑haystack” test inserts a key piece of information into a long document and measures whether the model can retrieve it. Primary evaluation metrics are:

Perplexity (PPL) on extended sequences.

Success rate of needle‑seeking (retrieval accuracy).

Technical Q&A Highlights

Q1: What is the physical meaning of rotary position encoding? The unit angle represents rotation speed; faster rotation encodes nearer tokens, slower rotation encodes farther tokens.

Q2: Why is extrapolation valuable? It enables models to process arbitrarily long texts. Because attention cost grows quadratically, practical systems often slice or chunk the input, but extrapolation reduces the performance drop when the window is extended.

Q3: Can BERT use relative position encoding? Yes, it is possible, but few works have explored it.

Q5: How is extrapolation loss measured? By comparing perplexity and needle‑seeking success before and after applying a method. Reducing the base angle slightly raises PPL, while enlarging the base has a minor impact on performance.