Can Mobius Break the Transformer Ceiling and Spark the Next Model Architecture Revolution?

Since the release of ChatGPT in 2022, the question of where the Transformer’s performance ceiling lies has been widely debated. Recent statements from former OpenAI inference lead Jerry Tworek and former Google Gemini pre‑training lead Rohan Anil claim that Transformers have reached their limit and now constitute the biggest bottleneck for AGI [1].

Domestic large‑model teams are experimenting with sparse or linear attention to lower computational complexity, achieving larger parameter scales under limited resources. However, it remains uncertain whether these lower‑complexity variants will also encounter a hard wall.

The Mobius team from Shanghai AI Lab proposes a possible next‑generation path: separating knowledge storage from reasoning to raise the capacity ceiling for continual learning and compositional generalisation, while also pressuring improvements in training and inference efficiency.

Three "sins" of the Transformer are identified:

Inability to quickly recall learned knowledge.

High cost to retain newly learned information.

Difficulty in composing unseen knowledge.

For (1), the author points out that long Chain‑of‑Thought (CoT) reasoning is inefficient and consumes excessive compute, because the model must traverse all layers to retrieve all knowledge. For (2), the classic continual‑learning problem is highlighted: adding new knowledge forces a brute‑force update of all parameters, despite the Transformer’s innate few‑shot and in‑context abilities. For (3), the limitation of compositional generalisation is discussed; while large Transformers can zero‑shot many tasks, they struggle with intuition‑driven domains such as scientific discovery.

The root cause is traced to hierarchical memory and reasoning: feed‑forward networks (FFN) store memory, attention layers (Attn) perform combination, and residual connections create a narrow pathway for knowledge flow. This hierarchy leads to incomplete knowledge retrieval, tangled storage, and diluted residual signals.

Mobius redesigns the architecture by extracting the memory of each layer into a shared pool that every attention layer can access. This design aims to solve the three sins:

More traversal opportunities increase the chance of retrieving critical knowledge, improving output relevance.

Flat storage lets new knowledge see the whole pool, reducing disruptive updates.

Simultaneous access enables multiple pieces of knowledge to be activated together, boosting compositional generalisation.

Experimental evidence shows that Mobius achieves nearly a 4× speedup in end‑to‑end inference compared with a standard Transformer while maintaining downstream task accuracy. With only 60 % of the training data, Mobius reaches MMLU scores comparable to a Transformer, and it doubles performance on specific compositional‑generalisation benchmarks. Relevant charts are included in the original article.

The author draws an analogy to the historic shift from RNN to Transformer: that transition turned a recurrent‑plus‑residual coupling into a full‑connection. Mobius performs a similar transformation, converting the recurrent‑plus‑residual coupling of knowledge into a full‑connection, potentially raising the model’s capacity ceiling.

Further discussion covers residual connections and latent reasoning. Mobius naturally supports backward residual connections and dynamic latent reasoning, allowing flexible information flow, higher knowledge compression, and more efficient iterative computation compared with traditional token‑by‑token CoT or static latent‑reasoning loops.

Looking ahead, the paper outlines three frontier directions where Mobius could have impact: recursive self‑evolution, AI‑assisted scientific discovery, and world‑model construction. The knowledge‑reasoning split aligns with the high‑density memory and flexible activation needs of these applications. Nevertheless, the author notes that hardware memory‑access and communication bottlenecks, as well as further attention‑mechanism redesigns, remain critical challenges for future Mobius versions.