Extending LLM Context to 1M Tokens: SAMBA, CoPE, RoPE, Retrieval Heads & Infini‑Attention

Transformer‑based large language models (LLMs) suffer from limited context windows, causing out‑of‑distribution (OOD) behavior when processing texts longer than the trained window. Challenges include degraded performance on complex tasks, attention dilution, and quadratic computational cost.

SAMBA: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

SAMBA combines hierarchical Mamba layers, SwiGLU, and sliding‑window attention to model sequences of unlimited length. The Mamba layer captures temporal dependencies, while a gating mechanism selects important inputs for long‑term memory. Sliding‑window attention ensures linear complexity, and multi‑scale perception layers provide non‑linear transformation and factual recall. Trained at 3.8 B parameters on 3.2 T tokens, SAMBA zero‑shots to 1 M tokens, reduces perplexity on the Proof‑Pile dataset, achieves perfect recall on 256 K‑token Passkey retrieval, and outperforms baselines on long‑context summarization.

Contextual Position Encoding (CoPE)

CoPE introduces a context‑dependent gating mechanism that decides which tokens contribute to distance measurement, computes cumulative gated sums to obtain fractional position values, and interpolates between nearest integer embeddings. This allows each query to measure distance using multiple units (e.g., token and sentence positions) and improves performance on selective copy, counting, language modeling, and code modeling tasks compared with traditional relative position encodings.

Base of RoPE Bounds Context Length

The paper analyzes how the base value of Rotary Position Embedding (RoPE) determines the maximum context length a model can handle. By deriving a theoretical lower bound (long‑term decay), it shows that larger training lengths require larger RoPE bases, independent of fine‑tuning strategies, and that using a base below this bound limits the model's ability to retrieve information from very long contexts.

Retrieval Head Mechanistically Explains Long‑Context Factuality

Retrieval heads are specialized attention heads that retrieve relevant information from long contexts. The authors define a retrieval score to measure copy‑paste behavior during autoregressive decoding and demonstrate that (1) all long‑context models possess a small subset (<5 %) of retrieval heads, (2) these heads exist even in models pretrained on short texts, (3) they activate dynamically based on context, and (4) ablating them drastically harms factual recall in tasks such as extractive QA and chain‑of‑thought reasoning.

Leave No Context Behind: Efficient Infinite Context Transformers with Infini‑Attention

Infini‑Attention introduces a compressive memory that enables Transformers to process unlimited input length with fixed memory. Input is split into fixed‑size segments; each segment undergoes local attention, then combines with a global attention computed using the stored memory M and a learned gating scalar β. The memory is updated with the current segment’s key‑value pairs for the next step. Experiments on language modeling, a 1 M‑token key‑retrieval benchmark, and a 500 K‑token book summarization task show state‑of‑the‑art performance with a 114× memory compression ratio.

Collectively, these works advance the ability of LLMs to handle extremely long contexts efficiently, offering linear‑time attention, context‑aware position encodings, and specialized retrieval mechanisms that together push the practical limits of language model applications.