Why Transformers Struggle with State Tracking and How Recurrence Could Fix It

DeepMind’s recent paper The Topological Trouble With Transformers argues that the core Transformer design cannot reliably track internal state across long dialogues or reasoning steps. The authors label this limitation “state tracking” and show that chain‑of‑thought (CoT) prompting merely prints hidden states to the output so they can be re‑ingested, which inflates computation and context usage.

Illustrative failures

Two concrete examples demonstrate the flaw. In a “guess‑the‑number” game (1‑100), the model must remember its secret number. Gemini 3 (Fast) contradicts itself: after saying the number is 42, it still answers “smaller” when the user guesses 42. A “bank” ambiguity test shows the model correctly identifies “river bank” in the first turn but later assumes a financial bank when asked about ATMs, because the disambiguation occurs in deep layers that later processing cannot see.

The paper uses the interpretability tool Patchscopes to reveal that semantic disambiguation for “bank” happens in layer 6, while subsequent tokens are processed only by layers 1‑5, which lack access to that information.

Why CoT is only a patch

CoT works by forcing the model to output its intermediate reasoning, effectively moving deep‑layer state to the surface where it can be read again. This reduces the state‑tracking problem but at a high cost: extra tokens consume the context window and increase inference latency.

Proposed direction: Re‑embracing recurrence

The authors suggest shifting focus from explicit CoT to implicit, recurrent dynamics. They categorize “recurrent Transformers” along two axes: the direction of recurrence (depth vs. sequence) and the number of tokens processed per recurrence step.

Depth‑wise recurrence (e.g., Looped Transformer, Universal Transformer) re‑uses the same layers multiple times but still pushes state deeper as sequences grow, only slowing the problem. True solution lies in sequence‑wise recurrence, where each new token receives the previous step’s state vector directly, mirroring classic RNN behavior while retaining attention mechanisms.

Recent state‑space models such as MAMBA, RWKV‑7, and DeltaNet exemplify this approach. An improved DeltaNet variant expands eigenvalue ranges into negatives, preserving parallel training benefits while surpassing standard Transformers on large‑scale language modeling benchmarks.

Future research avenues

Introduce recurrence at coarser granularity (e.g., sentence‑level loops).

Leverage residual connections for better representation alignment and lower training cost.

Adopt staged training: pre‑train a conventional feed‑forward model, then fine‑tune with recurrent mechanisms.

Implications for next‑generation models

The authors conclude that future foundational models must move beyond “repeatedly retrieving text” toward “fluid, continuously evolving representations” that maintain state across arbitrary time scales. Achieving such flowing memory is essential for stable, coherent long‑term cognition.