Looped Transformers: Recurrent Depth, Adaptive Recursion, Reasoning vs Memory

What This Loop Actually Is

The core idea of a looped transformer (recurrent depth) is to reuse the same stack of Transformer blocks multiple times instead of adding more distinct layers. Sebastian Raschka illustrates this with the recently open-sourced Nanbeige4.2-3B model: it has only 22 Transformer blocks but runs the input through them twice , yielding the equivalent of a 44-layer model while storing only 22 sets of weights. Block 23 reuses block 1's weights, block 24 reuses block 2's, and so on.

The trade-off ledger: parameters are roughly halved (embedding and output layers still take ~25% of Nanbeige's 3B parameters). However, forward and backward compute remain comparable to a 44-block model , and KV cache size is unchanged because intermediate states differ across passes. Nanbeige tried sharing KV cache across passes, which halved cache size but degraded performance; the released version uses independent caches. Training a 44-layer model then compressing it via looping works worse than training the looped structure from scratch. Empirically, 2 loops is the sweet spot ; 3+ loops give diminishing returns and training instability.

Thus recurrent depth is not "free deepening" but rather "using half the weight budget to buy full-depth compute."

Loop Count as a Flexible Knob

The number of loops can be dynamic, a concept dating back to the Universal Transformer (2018) , which introduced adaptive halting: a small learnable function emits a "halt probability" per token per step, accumulating until a threshold, allowing different tokens to undergo different numbers of loops—effectively allocating compute per token. ByteDance's Ouro-Thinking 2.6B pushes this further: 48 blocks looped 4 times (192 block applications) with a learned exit gate. Raschka notes an awkward detail: its HuggingFace implementation hard-codes the loop count by running all 4 passes first, then selecting the exit.

The 2025 Mixture-of-Recursions (MoR) hands the knob to a router: a small learned router examines each token's hidden state (including context) and decides how many recursive passes that token receives—so the same word at different positions can get different depths. Two routing modes: expert-choice (each step selects which tokens continue looping) and token-choice (assign each token a fixed path of 1, 2, or 3 passes upfront).

MoR experiments deliver a crucial lesson: at the smallest 135M scale, a vanilla Transformer outperforms MoR —looking only at small models would wrongly conclude recurrence is useless. Only at larger scales does MoR match or exceed baselines, especially under tight training compute budgets. This underscores that looped architectures must be validated at scale.

Recurrence Buys Reasoning, Not Memory

Two studies clarify what recurrence actually purchases. Zhu et al. (June 2025) separate memory and reasoning. In memory tasks (fixed parameter count), looping barely increases the model's ability to store and retrieve knowledge—capacity tracks independent parameter count. In reasoning tasks, reused blocks boost multi-step math performance without adding any parameters . The clean conclusion: recurrence is a compute mechanism, not a storage mechanism ; it buys thinking compute, not knowledge capacity.

SMELT (September 2026) scales this accounting to 54B non-embedding parameters using an MoE architecture. The middle half of blocks are applied twice, followed by three compensations: shrink hidden dimension to offset extra compute, increase expert count to restore total parameter count, adjust attention heads to keep KV cache comparable. Fitting scaling curves shows: to reach the same validation loss, the looped version needs 6.8% to 18% less training compute . In equal compute budgets, recurrence is a net win.

Between these, Geiping et al.'s "latent reasoning" model (3.5B params, 800B tokens) uses 4 shared blocks sandwiched between 2 initial and 2 final blocks. Each loop concatenates the initial block output with the previous hidden state, projects linearly, and feeds the shared blocks. Inference can use a fixed loop budget (8, 32, 64) or adaptive stopping via KL divergence between successive output distributions. Benefits vary by task: HellaSwag (common sense) saturates around 8 loops, while GSM8K and HumanEval (multi-step reasoning) genuinely benefit from more loops .

Is Hidden Chain-of-Thought a False Crisis?

Returning to the leak's other selling point: recurrence makes Astra's reasoning "harder to monitor." Astra remains a reasoning model trained with RLVR plus intermediate reasoning traces. Per Jensen Huang, the base model trained on ~ 100k Grace Blackwell GPUs ; OpenAI also bought tens of thousands of Mac minis and Mac Studios, but those serve only as RL interaction environments: task prompt → screenshot → model predicts mouse/keyboard actions → harness executes → new screenshot → repeat, using success signals for training—Astra's computer-use leap comes from this loop.

The controversy centers on token usage: Astra achieves comparable accuracy with notably fewer tokens than GPT-5.6 Sol . The Information interprets this as "reasoning hidden in architecture, monitoring regressed." Raschka disagrees: shorter chain-of-thought likely means the model makes fewer mistakes and backtracks less. He offers an apt analogy: a well-prepared student uses less scratch paper during an exam . Counter-evidence: the weaker Luna model uses ~80% more tokens than Sol at similar performance.

GPT-6 Astra, Looped Transformers, and Hidden Reasoning
https://magazine.sebastianraschka.com/p/gpt-6-astra-looped-transformers-and