Skip to content
From Attention to TTT: A New Linearization Path via Distribution Generalization

From Attention to TTT: A New Linearization Path via Distribution Generalization

This article derives a novel linearization path from Softmax Attention to Test-Time Training (TTT) by generalizing attention as a conditional expectation, applying three averaging approximations, and showing how DeltaNet and MesaNet emerge as special cases within this unified framework.

Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing

Generalized Definition

Given vectors, Softmax Attention is defined as:

o_i = \sum_{j=1}^{i} \frac{\exp(q_i^\top k_j)}{\sum_{l=1}^{i} \exp(q_i^\top k_l)} v_j

Each output requires pairwise computation with all previous keys and values, yielding quadratic complexity and KV Cache bottlenecks during decoding. Most linear attention variants essentially compress KV Cache. This work starts from KV Cache compression but first generalizes Softmax Attention by rewriting it as the expectation of a conditional distribution — the first meaning of the "averaging path": o_i = \mathbb{E}_{v \sim p(v|k_i)}[v] Taking a Dirac delta distribution p(v|k) = \delta(v - v_i) recovers the original Softmax Attention, so the generalization theoretically encompasses a broader function class.

Distribution Estimation

Using a Dirac delta means memorizing all key-value pairs. For compression, we replace it with a computable distribution: a conditional Gaussian where the mean predictor takes the current key as input with parameters, and is a preset hyperparameter:

p(v|k) = \mathcal{N}(v; \mu_\theta(k), \sigma^2 I)

The covariance matrix could also be generalized, but since only the mean vector matters ultimately, generalizing covariance adds no benefit. Under this choice, the output becomes the mean predictor's output: o_i = \mu_\theta(k_i) Assuming the mean predictor is efficient to compute, we no longer need to cache values — only keys — and can compute values on the fly, halving KV Cache.

Streaming Update

To estimate \mu_\theta, we treat incoming key-value pairs as streaming training data and use SGD to update the predictor. The natural objective is maximum likelihood, i.e., minimizing negative log-likelihood. One gradient descent step yields a recursive update rule, forming an RNN with the predictor parameters as state variables:

\theta_{t} = \theta_{t-1} - \eta 
abla_\theta \mathcal{L}(\theta_{t-1}; k_t, v_t)

The model continuously corrects its parameters to maximize key-to-value prediction accuracy, effectively compressing the incremental information of values relative to keys into finite-dimensional parameters. This recursive form is identical to Test-Time Training (TTT), except our starting point is compressing value information given keys to reduce KV Cache, without explicitly defining full write/read processes as in TTT.

Two Averages

After eliminating values, keys remain and the formulation still has Softmax Attention form with unchanged complexity. We apply a mean-field approximation: replacing the inner expectation with its mean, turning the core computation into a K=V Softmax Attention:

o_i \approx \sum_{j=1}^{i} \frac{\exp(q_i^\top k_j)}{\sum_{l=1}^{i} \exp(q_i^\top k_l)} \mu_\theta(k_j)

Further, assuming the value vectors are isotropic or their components orthogonal to the query average to zero, we can replace the query input with the mean key, or absorb the mean into the query, yielding a clean linear attention form:

o_i \approx \mu_\theta\left(\frac{\sum_{j=1}^{i} \exp(q_i^\top k_j) k_j}{\sum_{l=1}^{i} \exp(q_i^\top k_l)}\right) \approx \mu_\theta(\bar{k})

This produces a readout identical to TTT, which together with the streaming update forms an RNN with keys as inputs and parameters as state — full K and V no longer appear, having been progressively compressed into the parameters.

Result Reflection

The derivation involves three approximation steps:

Replacing exact value mapping with mean predictor — drops V Cache. The predictor's theoretical capacity is lower than exact pairwise matching, but streaming updates enhance token interactions; the author considers this a trade-off with no clear winner.

Mean-field approximation — mathematically concerning but architecturally just moves attention from output to input; in deep stacks this is likely not a fundamental defect.

Weighted average approximation to replace query input with mean key — fully linearizes attention to O(N) complexity. However, K=V attention is not always well approximated by the mean key; Multi-head Latent Attention (MLA) and DSV4 are essentially K=V MQA and clearly cannot reduce to mean key alone, or they would lose token interaction modeling. The biggest issue is this third step, an idealized simplification hard to satisfy in practice, necessitating periodic reintroduction of Softmax Attention — matching current hybrid approaches.

Linear Versions

In practice, the TTT mean predictor is often simplified to a linear model: \mu_\theta(k) = W k The streaming update rule becomes:

W_t = W_{t-1} + \eta (v_t - W_{t-1} k_t) k_t^\top

Adding a gating mechanism yields DeltaNet. Linearization enables parallelism and confines the model to a pure token-mixer role, leaving nonlinearity to MLP layers and stacking. For linear models, the mean-field approximation becomes an exact identity, not an approximation. Moreover, linear regression can be solved analytically without SGD:

W = \left(\sum_{t=1}^T v_t k_t^\top\right) \left(\sum_{t=1}^T k_t k_t^\top\right)^{-1}

Adding decay to the linear model's parameters recovers MesaNet:

W_t = \lambda W_{t-1} + \eta (v_t - W_{t-1} k_t) k_t^\top

Assuming MesaNet and DeltaNet share the same capacity ceiling, MesaNet's parameter updates reduce to Vanilla Linear Attention (VaLA), simpler than Delta Rule but at the cost of 50% larger state variables, indicating VaLA must increase state size to surpass Delta models.

Summary

This article uses a chain of averaging approximations to connect Attention, TTT, DeltaNet, and MesaNet. Following this derivation reveals the cost-benefit trade-offs at each step from Attention to TTT and various linear models, providing a more natural and general understanding of their relationships.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Attention MechanismLinear attentionTest-Time TrainingKV Cache compressionDeltaNetMesaNetSoftmax approximationTTT
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.