Why Transformers Are Naturally Succinct: Insights from the ICLR Best Paper

In a 2026 ICLR best‑paper, researchers conduct a deep theoretical examination of the Transformer architecture, arguing that it is a natural master of succinctness: it can describe complex concepts with orders‑of‑magnitude fewer symbols than many familiar models such as RNNs or finite automata.

A New Measure of Succinctness

Traditional analyses treat Transformers as language recognizers, but the paper shows that, given fixed‑precision hardware, a Transformer can only recognize a tiny fragment of regular languages—specifically a star‑free subset. For example, it cannot handle the pattern (aa)*, whereas classic RNNs easily cover all regular languages. The authors contend that, in the context of pursuing artificial general intelligence, this comparison misses the point and instead introduce succinctness from formal language theory as a more relevant metric.

The Super‑Counter Trick

To demonstrate the power of this succinctness, the authors devise a thought experiment where a polynomial‑size Transformer encodes solutions to the classic 2ⁿ tiling problem, whose computational complexity grows to EXPSPACE. They encode each row of the tiling as a string, and the Transformer's attention mechanism—using strict future masking—acts as a meticulous verifier, checking vertical constraints by comparing each tile with the one directly above it, while horizontal constraints are verified by a simple neighbor check. A built‑in “super counter” leverages the Transformer's parallel attention to count up to 2^(2^n), thereby locating every row and column precisely. Consequently, a modest‑size Transformer describes a solution whose minimal description length is double‑exponential, whereas equivalent LTL formulas or finite automata would explode exponentially.

When Minimalism Becomes a Burden

Although this extreme minimalism enables compact representation of intricate logic, it also makes verification daunting. The paper presents a stark example: determining whether a Transformer’s recognized language is empty—a seemingly basic question—turns out to be EXPSPACE‑complete. In the worst case, fully understanding whether a Transformer is merely “talking nonsense” could require double‑exponential resources, effectively impossible to compute. Additionally, the authors prove that any fixed‑precision Transformer can be translated into an equivalent LTL formula within exponential time, highlighting both the expressive strength and the verification difficulty.

Overall, the work opens a new research direction that evaluates Transformers by their succinctness rather than mere recognition ability, suggesting that future model design might benefit from considering description length as a guiding principle.