How Jeff Dean’s Vision Shaped Modern AI: From Neural Nets to Gemini

Key Observations

Scaling compute, data, and model size consistently improves performance.

Algorithmic and architectural innovations provide large gains.

The types of computation and hardware for AI are rapidly evolving.

Neural Networks and Back‑Propagation

Neural networks are the fundamental building blocks of modern AI. Back‑propagation optimizes network weights by propagating error gradients, enabling models to learn from data and generalize to new inputs.

Large‑Scale Distributed Training (DistBelief)

In 2012 Google trained a neural network 60× larger than any existing model using 16,000 CPU cores, achieving a ~70% boost on ImageNet. To support such scale they built DistBelief , a distributed system that splits models across machines (model parallelism) and replicates data across machines (data parallelism). Gradients were applied asynchronously, which is mathematically imperfect but proved effective.

Model Parallelism vs. Data Parallelism

Model parallelism partitions the network across devices, while data parallelism replicates the whole model on many devices and distributes different mini‑batches to each replica. Both strategies can be combined.

Specialized Hardware: TPUs

Google designed the Tensor Processing Unit (TPU) for neural‑network inference. The first generation used 8‑bit integer arithmetic and delivered 15–30× speedup and 30–80× energy efficiency over CPUs/GPUs. Subsequent generations (TPU v2, v3) and the latest Ironwood pod (9,216 chips, 4,614 TFLOPS per chip) scale both inference and training.

Open‑Source Frameworks

TensorFlow, JAX and the broader ecosystem (including PyTorch) provide flexible, high‑performance tools for research and production.

Transformer Architecture (2017)

The Transformer replaces sequential RNN processing with parallel self‑attention over all tokens, dramatically improving training efficiency and model quality. It underlies virtually all large language models.

Self‑Supervised Large‑Scale Language Modeling

Starting around 2018, researchers trained massive models on unlabeled text with objectives such as masked token prediction or next‑token prediction. Scaling data and model size yields continual performance improvements, giving rise to foundation models.

Sparse Mixture‑of‑Experts (MoE) Models

Google introduced sparse MoE models where only a small subset of expert sub‑networks are activated per token. This allows billions‑parameter models to be trained efficiently; experts specialize in domains such as dates, geography, or biology.

Software Abstraction: Pathways

Pathways is a scalable software stack that lets a single Python process address thousands of TPU chips, integrating with JAX for seamless parallelism. It abstracts hardware topology so developers write code once and run it on any scale.

Chain‑of‑Thought Prompting (2022)

Prompting models to generate intermediate reasoning steps (CoT) dramatically improves accuracy on tasks such as GSM8K math problems by using additional computation during inference.

Knowledge Distillation

Distillation transfers knowledge from a large “teacher” model to a smaller “student” model by matching the teacher’s output distribution, enabling compact models to achieve performance close to their larger counterparts.

Speculative Decoding (2023)

Speculative decoding combines a fast “drafter” model (10–20× smaller) with a large model. The drafter proposes multiple tokens which the large model then verifies, yielding a significant speedup in generation.

Impact and Outlook

Combining larger models, specialized hardware, and smarter software has dramatically raised AI capabilities. Continued investment and research are expected to further amplify model power, democratize expertise, and drive positive societal change, while emphasizing responsible stewardship.

References

Source video: https://video.ethz.ch/speakers/d-infk/2025/spring/251-0100-00L.html

Slides: https://drive.google.com/file/d/12RAfy-nYi1ypNMIqbYHjkPXF_jILJYJP/view

Code example

收
藏
，
分
享
、
在
看
，
给
个
三
连
击呗！