Oct 16, 2026 · Artificial Intelligence

STILL: Content-Aware Intra-Layer Hybrid Attention for Efficient LLM Linearization

STILL introduces a content-aware intra-layer hybrid attention mechanism that routes tokens to Softmax or linear attention based on self-saliency scores computed locally, preserving pre-trained norm structure via NP-Map, achieving 80% memory reduction and 83% prefill speedup with only 0.04B training tokens.

Efficient TransformersIntra-Layer Hybrid AttentionLLM Linearization

0 likes · 20 min read

STILL: Content-Aware Intra-Layer Hybrid Attention for Efficient LLM Linearization