Can 2‑Simplicial Attention Outperform Standard Transformers? A Deep Dive

Background

The 2017 paper Attention Is All You Need introduced the Transformer architecture, which underpins modern large‑language models. Empirical verification of the Transformer scaling law has driven rapid AI progress.

Motivation

Two practical bottlenecks remain: acquiring enough high‑quality tokens and using them efficiently. Improving the attention mechanism is a promising way to address these challenges.

2‑Simplicial Transformer

Clift et al. (2019) generalized dot‑product attention to a trilinear form, called the 2‑simplicial Transformer. In addition to the standard projection matrices W_Q, W_K, and W_V, two extra matrices W_K′ and W_V′ are introduced:

K′ = X W_K′
V′ = X W_V′

The attention logit is computed as a three‑way product of query Q, key K, and the extra key K′: logit_{i,j,k} = \langle Q_i, K_j, K′_k \rangle This yields a third‑order tensor of logits. The output is a weighted sum of the value tensors, analogous to standard attention but using the trilinear logits.

Rotation‑Invariant Formulation

Rotary Position Embedding (RoPE) rotates queries and keys to encode relative positions while preserving inner products under orthogonal transforms. A naïve trilinear extension of RoPE is not rotation‑invariant. Meta identified determinant‑based functions that remain invariant under rotation, enabling a rotation‑invariant trilinear attention.

Complexity and Model Design

The naïve 2‑simplicial attention has O (n³) time and memory complexity, which is impractical for long sequences. Meta parametrised the operation as O (n·w₁·w₂) by restricting attention to sliding windows of size w₁ (for K) and w₂ (for K′). Each query attends only to a local region, dramatically reducing cost.

Empirical evaluation of various window configurations identified (w₁, w₂) = (512, 32) as a sweet spot: the resulting compute cost matches that of a standard dot‑product attention with a 48 k context window.

Scaling‑Law Analysis

The standard scaling‑law formulation is: L(N, D) = E·N^{‑a} + B·D^{‑b} + C Meta fitted the coefficients on their models and obtained a ≈ 0.49 and b ≈ 0.5, confirming the established token‑to‑parameter proportionality. Under a fixed token budget, the 2‑simplicial Transformer exhibits a steeper loss‑vs‑parameter slope (larger α) than a standard dot‑product Transformer, indicating more efficient token utilisation.

Experimental Results

Meta trained a suite of mixture‑of‑experts (MoE) models ranging from 1 B active parameters (57 B total) to 3.5 B active parameters (176 B total). Negative log‑likelihood improves monotonically with model size, but models below 2 B active parameters do not benefit from the 2‑simplicial attention.

Conclusions and Limitations

The rotation‑invariant 2‑simplicial attention can achieve a higher scaling‑law exponent, suggesting more efficient token utilisation under a fixed compute budget. However, its cubic‑order operations require windowing tricks, and the performance gains disappear for models smaller than roughly 2 B active parameters. Further research is needed to generalise the approach and reduce overhead.

Reference

Paper: Fast and Simplex: 2‑Simplicial Attention in Triton (arXiv:2507.02754)

PDF: https://arxiv.org/pdf/2507.02754.pdf