Can a Pre‑1930 Language Model Infer Einstein’s Relativity? Insights from the Talkie‑1930 Project

The team led by Alec Radford, David Duvenaud, and Nick Levine trained a 13B parameter language model, Talkie‑1930, exclusively on English texts published before 1931 (≈2.6 trillion tokens) to create a clean historical baseline.

After training, they opened a 24‑hour live channel where Claude Sonnet 4.6 conversed with Talkie‑1930, publishing the dialogue logs for public inspection.

To assess whether a model limited to pre‑1930 knowledge can “anticipate” future events, they sampled ~5,000 historical‑event descriptions from the New York Times “On This Day” column and measured each description’s surprisal (information‑theoretic surprise) under Talkie. As expected, events before 1930 yielded low surprisal, while post‑1930 events showed a sharp increase, peaking in the 1950s‑60s and then plateauing.

Motivated by DeepMind founder Demis Hassabis’s question about models inferring unseen knowledge, they examined whether a sufficiently large model could deduce concepts such as Sikorsky’s 1935 helicopter patent, Turing’s 1936 computability paper, or Carlson’s 1942 xerography patent—items Talkie could not have directly seen.

They also used the HumanEval programming benchmark to probe “pollution” effects. Talkie, unaware of modern code, was given a few Python function examples and asked to generate new solutions. Results showed modest but consistent improvement with scale; successes were limited to trivial one‑line programs or minor variations (e.g., decoding a rotation cipher by swapping ‘+’ with ‘‑’), suggesting an emerging grasp of abstract concepts like inverse functions.

To isolate the impact of training‑data diversity, they trained a “modern twin” model with identical architecture but modern web data (FineWeb). Comparing Talkie and its twin across language understanding, numeric computation, and knowledge domains revealed Talkie lagging overall, yet after filtering out “out‑of‑scope” questions (those requiring post‑1930 knowledge), the performance gap roughly halved.

The authors identified two major obstacles in building historical models: (1) time leakage —later annotations, re‑prints, or digitization notes inadvertently introduce post‑1930 information, causing the model to answer anachronistic questions (e.g., naming Roosevelt for 1936). They built an n‑gram‑based anomalous‑word detector to filter such leaks, though it remains imperfect. (2) data quality —OCR errors on scanned books dramatically reduce token fidelity; models trained on raw OCR text performed at ~30 % of the quality of those trained on manually transcribed text, improving to ~70 % after regex cleaning.

For instruction tuning, they avoided modern dialogue data that would “contaminate” the model’s vintage voice. Instead, they generated instruction–response pairs from historical etiquette manuals, letters, recipes, and encyclopedias, using Claude Opus 4.6 as the user and Talkie as the assistant, with Claude Sonnet 4.6 as the judge. Scores rose from an average of 2/5 to 3.4/5 after this post‑training process.

The researchers acknowledge that using a modern judge introduces its own era bias and plan to evaluate Talkie with self‑judgment in future work. They are currently scaling up to a GPT‑3‑class model with >1 trillion historical tokens, aiming for a GPT‑3.5‑level capability comparable to early ChatGPT.