How GPT-3 Evolved: From Transformer Roots to Massive Language Models

In 2017, the Google Machine Translation team introduced the Transformer in the paper “Attention Is All You Need,” discarding RNNs and CNNs and relying solely on attention mechanisms, establishing the encoder‑decoder architecture for machine translation.

In June 2018, OpenAI released GPT‑1 (“Improving Language Understanding by Generative Pre‑Training”) with 117 million parameters, using only the Transformer decoder for feature extraction and pre‑training on large English corpora such as Wikipedia and WebText.

In October 2018, Google published BERT (“Pre‑training of Deep Bidirectional Transformers for Language Understanding”), employing the Transformer encoder with 110 million‑to‑1.3 billion parameters, achieving superior performance on many NLP tasks compared to GPT.

In February 2019, OpenAI introduced GPT‑2 (“Language Models are Unsupervised Multitask Learners”) with 1.5 billion parameters, again using the decoder‑only design and matching BERT’s performance.

In June 2020, OpenAI unveiled GPT‑3 (“Language Models are Few‑Shot Learners”) with 175 billion parameters, continuing the decoder‑only approach and delivering striking text‑generation capabilities.

Subsequent milestones include OpenAI’s Codex (July 2021) for code generation, Instruct‑GPT (2022) trained with human feedback, and the launch of ChatGPT (late 2022) built on GPT‑3.5, which became a phenomenon.

OpenAI initially did not anticipate using the decoder for language modeling; the discovery emerged during exploration. Although Google invented the Transformer, it did not capitalize on it first, allowing OpenAI to leverage the architecture for GPT, illustrating that the impact of a technology depends more on effective utilization than on being the original inventor.