Skip to content
Tagged articles

Softmax approximation

1 articles · Page 1 of 1
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Nov 13, 2026 · Artificial Intelligence

From Attention to TTT: A New Linearization Path via Distribution Generalization

This article derives a novel linearization path from Softmax Attention to Test-Time Training (TTT) by generalizing attention as a conditional expectation, applying three averaging approximations, and showing how DeltaNet and MesaNet emerge as special cases within this unified framework.

Attention MechanismDeltaNetKV Cache compression
0 likes · 11 min read
From Attention to TTT: A New Linearization Path via Distribution Generalization