Geoff Hinton on Scaling Laws, Multimodal AI, and the Future of Intelligence

Joel Hellermark interviews AI pioneer Geoff Hinton, reviewing his career and discussing key topics such as neural networks, scaling laws, multimodal learning, fast weights, and AI ethics.

Early Education and Career

Hinton began at Cambridge studying physiology, became disillusioned, turned to philosophy, and finally pursued a Ph.D. in artificial intelligence at Edinburgh, graduating in 1978.

Hellermark: "Back at Cambridge, you tried to understand the brain. What was that like?" Hinton: "It was very disappointing. Physiology only taught how neurons fire, not how the brain works. Philosophy was the same. AI let me simulate and test ideas."

Influences and Carnegie Mellon

Donald Hebb’s work on connection strength and John von Neumann’s book sparked his interest. At Carnegie Mellon (1982‑1987) he encountered a Lisp machine, wrote programs, and coined the term “hidden layer” after learning about hidden Markov models.

Hinton: "That’s why the term ‘hidden layer’ exists in neural networks."

Collaboration with Ilya Sutskever

Hinton recalls Ilya’s eager knock on his lab door, their early work on back‑propagation, and the intuition that larger models perform better—a scaling‑law insight later confirmed by GPT‑4.

Hinton: "Ilya thought bigger models would work better. I thought it was an excuse, but it turned out to be true."

Scaling Laws and Creativity

Hinton emphasizes that increasing data and compute drives AI performance and creativity. He illustrates GPT‑4’s ability to draw analogies, such as comparing compost piles to atomic bombs via shared chain‑reaction concepts.

Hinton: "GPT‑4 sees the common structure between seemingly unrelated things, which fuels its creativity."

Cognitive Perspectives

He outlines three views of cognition: symbolic (logic‑based), vector (all information as vectors), and embedding (symbols converted to embeddings that interact to predict the next symbol). He argues that large language models operate like the embedding view.

Multimodal Learning

Hinton argues that combining language, images, video, and sound will dramatically improve reasoning and reduce reliance on textual data.

Hinton: "Multimodal systems understand space better and need less language to learn."

Simulated Computing vs. Digital

He discusses attempts at low‑power analog computing for AI, noting digital weight sharing is far superior for knowledge transfer.

Fast Weights and Temporal Scales

Hinton introduces the concept of fast weights—temporary synaptic changes that enable rapid, context‑dependent memory, a mechanism absent from current neural models due to batch processing constraints.

Hinton: "The brain uses fast weights for short‑term storage; we need architectures that can do the same."

AI Ethics and International Competition

He warns of potential harms (autonomous weapons, surveillance) and notes the relentless AI race between the United States and China, dismissing calls for a pause.

Future Outlook

Hinton sees healthcare, new material discovery, and understanding brain learning (whether the brain performs back‑propagation) as major future directions.

Hinton: "The biggest unanswered question for me is whether the brain does back‑propagation."

Overall, the interview provides deep insights into the evolution of AI research, the importance of scaling, multimodal integration, and the ethical responsibilities of the community.