Dropping Intermediate Tokens: How Prefix Sliding Achieves Up to 3× Faster Long-Context Reasoning

Attention pattern in long reasoning

Full‑attention Transformers incur a per‑step cost that grows linearly with the length of the reasoning chain. Analysis of a Qwen‑3‑1.7B trajectory on the AIME‑25 benchmark shows a “two‑head high, middle low” attention distribution: the initial task prefix and the most recent tokens receive high attention, while tokens in the middle receive low attention.

Prefix Sliding method

Prefix Sliding keeps the task prefix (system prompt, task description, tool information) permanently in the KV cache and maintains a sliding window of the most recent W generated tokens. As generation proceeds, tokens older than the window are removed from the cache, so the cache size never exceeds prefix_length + W.

Example: with a 100‑token prefix and W = 4096, a 100 000‑token trajectory requires only about 4 196 tokens in the KV cache. Once the window is full, the per‑step cost stops increasing.

Implementation details

On NVIDIA Hopper GPUs the authors implement a custom FlashAttention kernel that skips tiles completely outside the active region and masks tiles that partially overlap the active region, achieving speed close to a naïve sliding‑window implementation.

Evaluation without additional training

Using Qwen‑3‑1.7B without further fine‑tuning, the method is evaluated on AIME‑25, GPQA, and MATH‑500. Accuracy remains comparable to full attention, while throughput at a 128 K token length reaches 5 224 tok/s versus 448 tok/s for full attention. The reported “up to 3× speedup” refers to equal reasoning time, not raw token‑per‑second ratios.

Reinforcement‑learning extension

In the RL stage, Prefix Sliding enables rollouts longer than 100 k tokens. To avoid back‑propagating through the entire trajectory, truncated back‑propagation is used. KL divergence drops sharply when the window is enlarged to 2×, further improves at 4×, and stabilises at 8× the base window.

Comparison with alternatives

Last‑k : repeatedly processes retained tokens, leading to higher compute.

Plain sliding window : eventually discards the prefix, losing task information.

Summary : requires generating an extra summary and introduces additional hyper‑parameters; the model may ignore the summary and recompute.

Prefix Sliding : retains the prefix, slides only intermediate tokens, and achieves the best overall performance‑efficiency trade‑off.

Effective receptive field

The theoretical receptive field of a window spanning W tokens across L layers is W × L, but cited analysis indicates the practical effective receptive field is about 1.5 × W. Consequently, when back‑propagating through the last window only a limited context is needed.

Limitations

Short tasks (e.g., HealthBench with average generation ≈ 2 086 tokens) see modest gains because the 2 048‑token window is already saturated.

The KV‑cache cost for very long prefixes during the pre‑fill phase is unchanged.

Multi‑turn interactions raise the question of whether user instructions should remain in the permanent prefix or be allowed to slide out; the paper does not resolve this.

Empirical study is limited to standard Transformers; other architectures or hybrid window schemes are not evaluated.

Future directions

Prefix Sliding reframes long‑reasoning efficiency as a context‑management problem: deciding how long generated history should be retained. Further work is needed to devise finer‑grained policies for different domains, validate on larger models, and explore memory mechanisms for code‑generation, agent, and multi‑turn scenarios.

Paper: https://arxiv.org/abs/2608.26070 Code:

https://github.com/Muennighoff/prefix-sliding