Can Large Language Models Get Stronger Without Human Language Training? A New Pre‑Pre‑Training Path

Hypothesis

The authors hypothesize that the structural richness of data, rather than its semantic content, is the key factor enabling large‑scale language model training. If true, any data source with comparable structural properties could serve as a pre‑training substrate.

Neural Cellular Automata (NCA) as a synthetic data source

NCA are a learned extension of Conway’s Game of Life. Each NCA instance is a neural network that defines a latent state‑transition rule; when run on a 2‑D grid it produces large‑scale spatiotemporal patterns. The grid is divided into 2×2 patches (the same patching used by visual Transformers) and each patch is tokenized, yielding a sequence of tokens that the Transformer predicts autoregressively.

Key properties of NCA data:

Heavy‑tailed token frequency distribution and Zipf‑like statistics similar to natural language.

Long‑range dependencies emerging from the underlying dynamics.

Each sampled NCA encodes a unique latent rule, forcing the model to infer the rule from context for accurate next‑token prediction.

Pre‑pre‑training protocol

Training proceeds in three stages:

Pre‑pre‑training on a synthetic dataset (NCA, or baselines C4 or Dyck).

Standard language‑model pre‑training on a natural‑language corpus.

Downstream fine‑tuning on specific tasks (web‑text, mathematics, code).

Experimental setup

All regimes use a fixed token budget of 1.64 × 10⁸ tokens for the pre‑pre‑training phase. The Transformer architecture and hyper‑parameters are kept identical across experiments.

Four regimes are compared:

Training from scratch (no pre‑pre‑training).

Pre‑pre‑training on natural text (C4).

Pre‑pre‑training on a synthetic Dyck language.

Pre‑pre‑training on NCA data (proposed method).

Results under equal token budget

Across web‑text, mathematics, and code benchmarks, the NCA‑pre‑trained models achieve:

Up to 6 % lower perplexity than the best baseline.

Approximately 40 % faster convergence (≈1.4× speed‑up).

Stronger reasoning performance on downstream tasks.

Scaling natural‑text pre‑pre‑training

When the C4 pre‑pre‑training budget is increased ten‑fold to 1.6 billion tokens** while keeping the NCA budget at 1.64 × 10⁸ tokens, the NCA models still dominate:

Convergence speed improves by a factor of 1.4× .

Final perplexity is reduced by roughly 5 % .

Analysis of transferable components

Re‑initialization experiments show that attention layers carry the most transferable computation primitives, whereas MLP layers encode domain‑specific knowledge that transfers only when source and target tasks align.

Complexity matching is crucial: simpler NCA dynamics benefit code tasks, while more intricate dynamics improve performance on mathematics and web‑text tasks.

Why NCA data transfer better

Because NCA sequences contain no semantic shortcuts, every token forces the model to perform in‑context rule inference. This encourages the emergence of “induction heads” – attention circuits that copy patterns from early positions to later ones – a mechanism previously linked to strong in‑context learning.

Implications for training design

Controlling the structural complexity of synthetic data provides a new degree of freedom for aligning pre‑training signals with downstream objectives. For example, designers can select simpler NCA rules for code‑related tasks or richer dynamics for domains such as genomic‑sequence modeling.

The study suggests a shift from asking whether synthetic pre‑training works to exploring how far it can be pushed: foundational models can first acquire reasoning abilities from pure synthetic data and then fine‑tune on a modest, carefully curated natural‑language corpus, potentially reducing inherited textual biases.

Reference (arXiv): https://arxiv.org/pdf/2603.10055

Blog post with additional details: https://hanseungwook.github.io/blog/nca-pre-pre-training/