How Princeton’s DYSCO Decoder Boosts Long-Context Reasoning by 25% Without Fine‑Tuning

Background

Large language models (LLMs) advertise context windows of 128 K tokens or more, but empirical evaluation shows a sharp accuracy decline—named “Context Rot”—as input length increases, even on simple reasoning tasks.

DYSCO Algorithm

DYSCO (Dynamic Attention‑Scaling Decoding) is a training‑free decoding method that dynamically rescales attention during generation. It operates in three synchronized stages:

1. Aggregation

At each generation step a partial forward pass extracts the attention distributions of a predefined set of query‑optimized retrieval heads (QRHEAD). The scores from these heads are averaged to produce a relevance score for every token in the current context. A momentum term smooths relevance scores across steps. For the first step, relevance is initialized by averaging the attention of the last prompt token.

2. Selection

Tokens are ranked by cumulative relevance. A top‑p‑like criterion selects the smallest token set whose cumulative probability exceeds a threshold, subject to a hard cap (e.g., 8192 tokens) to bound intervention size.

3. Rescaling

An intervention vector adds a positive log‑bias to the selected tokens and zero to all others. This bias is added to the attention logits before the softmax, explicitly amplifying the importance of the chosen tokens. The modified attention is then used for the full forward pass to sample the next token.

Because QRHEADs are typically located in middle layers (e.g., layers 17‑20 of Qwen‑3‑8B), the partial forward pass can stop early, keeping extra cost low. On a 128 K input with an 8 K output, DYSCO adds only ~3.8 % FLOPs.

Experimental Validation

Benchmarks were run on Qwen‑3‑4B/8B/32B and Llama‑3.1‑8B‑Instruct, comparing DYSCO against vanilla decoding, YaRN, and uniform attention scaling (UNIATTNS).

PATH TRAVERSAL synthetic task: Baseline accuracy drops from ~60 % at 4 K context to <20 % at 16 K. DYSCO maintains higher scores; at 32 K tokens Qwen‑3‑32B improves from 21 % to 33 % accuracy, outperforming UNIATTNS.

Long‑context reasoning suites (MRCR, LONGBENCHV2, CLIPPER): DYSCO consistently yields gains. Combined with YaRN, Qwen‑3‑8B’s LONGBENCHV2 score rises by 29 % and MRCR by 22 %.

Recall‑oriented tasks (HotpotQA, InfBench): Stable improvements are observed, and DYSCO outperforms retrieval‑augmented generation (RAG) and prompt‑compression methods, especially beyond 64 K context.

Ablation Studies

Replacing QRHEAD with randomly chosen heads degrades performance, confirming the necessity of targeted retrieval heads.

Static scaling of the selected tokens (no dynamic update) lags behind full dynamic rescaling, highlighting the importance of continuously shifting attention focus.

Conclusion

DYSCO demonstrates that extracting and amplifying signals from a small set of retrieval‑oriented attention heads during decoding can effectively mitigate the “Context Rot” problem. It provides a lightweight, plug‑and‑play solution for long‑context inference with minimal computational overhead.

Paper: DYSCO: Dynamic Attention‑Scaling Decoding for Long‑Context LMs – https://arxiv.org/pdf/2602.22175

Code: https://github.com/princeton-pli/DySCO