Understanding GPT: Word Vectors, Transformers, and Model Architectures (GPT‑2, GPT‑3)

ChatGPT, a widely known chatbot from OpenAI, has sparked great interest; this article aims to demystify the mathematical and architectural foundations behind GPT models.

It begins with word vectors, showing how each token (e.g., the word "king") can be represented by a high‑dimensional numeric vector; an example 50‑dimensional vector is displayed below.

[0.50451 , 0.68607 , -0.59517 , -0.022801, 0.60046 , -0.13498 , -0.08813 , 0.47377 , -0.61798 , -0.31012 , -0.076666, 1.493 , -0.034189, -0.98173 , 0.68229 , 0.81722 , -0.51874 , -0.31503 , -0.55809 , 0.66421 , 0.1961 , -0.13495 , -0.11476 , -0.30344 , 0.41177 , -2.223 , -1.0756 , -1.0783 , -0.34354 , 0.33505 , 1.9927 , -0.04234 , -0.64319 , 0.71125 , 0.49159 , 0.16754 , 0.34344 , -0.25663 , -0.8523 , 0.1661 , 0.40102 , 1.1685 , -1.0137 , -0.21585 , -0.15155 , 0.78321 , -0.91241 , -1.6106 , -0.64426 , -0.51042 ]

Similarity between vectors can be measured with cosine similarity, illustrated with a 5‑dimensional personality example where the most similar vectors are identified.

The core of modern language models is the Transformer architecture, which processes token vectors through an encoder‑decoder pipeline. Within each layer, self‑attention creates three projections—query (Q), key (K), and value (V)—by multiplying the input vectors with learned weight matrices. Scores are obtained by dot‑product of Q and K, normalized with softmax, and then used to weight V, producing contextualized representations (Z). These are concatenated and passed through a feed‑forward network before moving to the next layer.

GPT‑2 adopts a decoder‑only stack, using masked self‑attention so that each token attends only to previous tokens. A concrete example with the phrase "robot must obey orders" demonstrates how queries, keys, and values are scored, masked, normalized, and combined to generate the next token probabilities.

GPT‑3 scales this design dramatically, employing 96 decoder layers and 175 billion parameters. Its operation still follows three steps: vectorization of input tokens, extensive matrix multiplications (including self‑attention and feed‑forward), and projection of the final vectors back to token probabilities.

The article concludes that while these models are still emerging, understanding their underlying mathematics and architecture provides a solid foundation for future AI applications.