Oct 16, 2026 · Artificial Intelligence

STILL: Content-Aware Intra-Layer Hybrid Attention for Efficient LLM Linearization

STILL introduces a content-aware intra-layer hybrid attention mechanism that routes tokens to Softmax or linear attention based on self-saliency scores computed locally, preserving pre-trained norm structure via NP-Map, achieving 80% memory reduction and 83% prefill speedup with only 0.04B training tokens.

Efficient TransformersIntra-Layer Hybrid AttentionLLM Linearization

0 likes · 20 min read

STILL: Content-Aware Intra-Layer Hybrid Attention for Efficient LLM Linearization

AIWalker

Feb 26, 2025 · Artificial Intelligence

Why Linear Attention Lags Behind Softmax and How Two Simple Tweaks Close the Gap

The paper analytically identifies injectivity and local modeling as the two key factors causing the performance gap between linear and Softmax attention, proposes the InLine attention modifications to restore these properties, and demonstrates through extensive Vision Transformer experiments that the enhanced linear attention matches or surpasses Softmax while retaining linear computational cost.

Attention MechanismEfficient TransformersSoftmax Attention

0 likes · 24 min read