Can 2‑Simplicial Attention Redefine Transformer Scaling Laws?

Background and Motivation

Since the seminal 2017 "Attention Is All You Need" paper, the Transformer architecture has become the foundation of modern language models. Scaling laws linking model size, token count, and loss have driven rapid progress, but acquiring sufficient high‑quality tokens remains a bottleneck.

2‑Simplicial Attention

Meta’s new paper, Fast and Simplex: 2‑Simplicial Attention in Triton (arXiv:2507.02754), extends the traditional dot‑product attention to a rotation‑invariant trilinear form. By generalizing RoPE to a signed‑determinant based operation, the authors achieve the same expressive power as a 2‑simplicial Transformer while preserving rotational invariance.

The trilinear attention logit is computed as a three‑way tensor product of query Q, key K, and an additional projected key K'. The resulting attention tensor is then normalized with softmax to produce weights for aggregating values.

Rotational Invariance

Standard RoPE applies a rotation R to queries and keys, preserving the inner product <q_i, k_j>. However, the trilinear form is not inherently rotation‑invariant. The authors introduce a signed determinant operation to construct a rotation‑invariant trilinear function, enabling the use of RoPE‑style positional encoding in the 2‑simplicial setting.

Complexity and Model Design

The naïve 2‑simplicial attention has O(n³) complexity, which is impractical for long sequences. Meta mitigates this by parameterizing attention as O(n × w₁ × w₂), where w₁ and w₂ define local sliding windows for K and K'. This reduces computation while retaining the benefits of the trilinear form.

Complexity comparison:

Standard causal dot‑product attention: O(n²) (two matrix multiplications per layer).

2‑Simplicial attention with windows (w₁, w₂): O(n × w₁ × w₂), adding a single extra multiplication due to the trilinear einsum.

Meta selects window sizes (512, 32), achieving computational cost comparable to a 48k context length dot‑product attention.

Experimental Evaluation

The authors train a series of Mixture‑of‑Experts models ranging from 1 B active parameters to 3.5 B active parameters (up to 176 B total parameters). Results show:

Negative log‑likelihood scaling improves as model size increases.

For models under 2 B active parameters, 2‑simplicial attention offers no benefit.

Scaling‑Law Coefficients

By fitting loss curves, the authors estimate scaling‑law exponents (α) and intercepts (β) for both standard and 2‑simplicial Transformers. The 2‑simplicial variant exhibits a steeper α, indicating a higher exponent in the scaling law, and a more favorable β, suggesting better token efficiency.

Conclusions

The rotation‑invariant 2‑simplicial attention mechanism can surpass standard dot‑product attention in scaling‑law performance, especially for larger models where token budgets are limited. However, its benefits are not evident for smaller models, and the cubic complexity necessitates careful windowed implementations.