How a 1930‑Era AI Model Without Any Computer Knowledge Learned to Write Python

Model description

talkie-1930-13b is a 13‑billion‑parameter language model trained on 2.6 trillion tokens of English text published before 1 January 1931. The corpus consists of books, newspapers, journals, and scientific magazines, selected to respect the U.S. copyright public‑domain cutoff of 1930.

Temporal awareness experiment

To measure the model’s awareness of its temporal knowledge gap, the authors fed ~5,000 historical events from the New York Times “On This Day” archive and recorded the model’s “surprise” (negative log‑likelihood) for each event. The curve shows low surprise for pre‑1930 events, a gradual increase immediately after 1930, and a sharp spike during the 1950‑60s when transistor and television technologies appear.

Programming capability test

The model was evaluated on the OpenAI HumanEval benchmark. Prompts included a few example Python functions, after which talkie‑1930 generated a correct solution for a task that required changing a constant +5 to -5 in an encryption function—only a single character edit.

A twin model, talkie‑web‑13b, trained on modern internet data with the same architecture, solved more tasks, illustrating a scaling‑law trend: larger models solve more programming problems.

Standard LLM evaluation

When both models were run on a suite of language‑understanding, reasoning, and knowledge tasks, talkie‑1930 lagged overall but the gap halved after removing items that require post‑1930 knowledge (e.g., internet or DNA). On core language‑understanding and mathematical reasoning tasks the two models performed comparably, suggesting those abilities are not heavily dependent on modern data.

Analysis of remaining gap

The authors attribute the residual performance difference to two factors:

Poor OCR quality of scanned 1930s newspapers, which introduces noise into the training corpus.

Divergent topic distribution: the vintage corpus contains more cooking and etiquette material and less technology.

Conversational fine‑tuning

To turn the model into a chat assistant without injecting 21st‑century language styles, the team curated a “natural dialogue” dataset from pre‑1930 etiquette manuals and letter‑writing guides. They used Claude Sonnet 4.6 as a teacher in a reinforcement‑learning‑from‑human‑feedback (RLHF) pipeline to generate instruction‑following data and fine‑tuned a 7‑b version.

After fine‑tuning, the model adopted modern list‑style formatting (e.g., “1. 2. 3.”), a style present in the teacher model but absent from the vintage source data. This style leakage demonstrates that RLHF can re‑introduce contemporary stylistic biases.

Future direction

The authors propose eliminating teacher‑model bias by enabling the vintage model to self‑teach, thereby preserving its historical linguistic style while retaining downstream utility.

Resources

Project page: https://talkie-lm.com/introducing-talkie

Model repository: https://huggingface.co/talkie-lm

Chat interface: https://talkie-lm.com/chat