Can the ‘Canon’ Layer Unlock New Limits in Large Language Models?

Introduction

The paper proposes a lightweight Canon layer – a trainable 1‑D convolution with kernel size 4 – to enhance horizontal information flow between neighboring tokens in large language models (LLMs). By inserting this layer at different points in the network, the authors can isolate and measure core cognitive abilities such as inference depth, inference breadth, and knowledge capacity.

Canon Layer Variants

Canon‑A : placed before the attention block.

Canon‑B : embedded inside the attention mechanism.

Canon‑C : inserted before the MLP (feed‑forward) block.

Canon‑D : integrated within the MLP block.

All variants compute a weighted combination of a token’s immediate neighbours and add the result via a residual connection, requiring only a few extra FLOPs and no architectural redesign.

Experimental Setup

Five synthetic pre‑training tasks are designed to evaluate specific capabilities while removing noise from real‑world data. The authors benchmark the following model families:

Standard Transformers (with RoPE, ALiBi, hybrid‑ALiBi, or NoPE positional encodings).

Linear‑attention models (GLA – Generalized Linear Attention).

State‑space models (Mamba and its upgraded version Mamba2).

Each architecture is evaluated with and without the Canon layer in each of the four insertion points.

Key Findings

Adding a Canon layer increases inference depth by 200‑400 % and inference breadth by ~30 % with negligible computational overhead.

For Transformers without positional encodings (NoPE), Canon restores performance to the level of RoPE‑augmented models.

Linear‑attention models equipped with Canon achieve performance comparable to Mamba2 on the Brevo memory‑intensive benchmark.

Linear models still lag behind Transformers on complex reasoning tasks, while Mamba2 excels on tasks requiring long‑range memory.

Removing the 1‑D convolution from Mamba2 degrades it to the level of gated linear attention, highlighting the importance of horizontal token flow.

Real‑World Pre‑Training

A 13‑billion‑parameter model is trained on 1 trillion tokens with a context length of 4096. Although statistical significance is limited by noise, the same trends observed in synthetic tasks appear: the Canon layer consistently improves performance across all tested architectures.

Analysis and Implications

The authors argue that many architectural bottlenecks stem from inefficient token compression and retrieval rather than raw memory capacity. The controllable synthetic benchmark enables precise isolation of intrinsic biases and facilitates systematic architecture comparison. Future work aims to combine the Canon layer with higher‑quality data pipelines and reinforcement‑learning‑based fine‑tuning to unlock deeper hierarchical reasoning.

Conclusion

The Canon layer provides a simple, low‑cost mechanism for improving horizontal information flow in LLMs, leading to substantial gains in inference depth, breadth, and scalability across diverse model families.

Code example

收
藏
，
分
享
、
在
看
，
给
个
三
连
击呗！