How ChatGPT Works: An In‑Depth Explanation by Stephen Wolfram

Stephen Wolfram, the creator of the Wolfram Language, writes a detailed exposition of the mechanisms behind ChatGPT and large language models (LLMs). He frames the discussion as a first‑person narrative and promises a “easter egg” at the end.

Adding One Word at a Time

ChatGPT generates text by repeatedly predicting the next token (often a word or part of a word) based on the statistical patterns learned from billions of web pages and books. The model does not simply copy text; it matches meaning and selects a token from a probability‑ranked list.

The selection can be deterministic (always picking the highest‑probability token) or stochastic, controlled by a temperature parameter (commonly set to 0.8) that introduces randomness and yields more interesting, less repetitive output.

For demonstration, Wolfram uses a lightweight GPT‑2 model that can run on a standard desktop. The article includes several Wolfram‑language code snippets (shown as images) that retrieve the underlying language‑model neural network, request the top‑5 token probabilities, and iteratively apply the model to generate text.

When the temperature is set to zero, the model quickly falls into repetitive or nonsensical loops. Introducing randomness (temperature = 0.8) produces varied continuations, as illustrated by multiple example outputs.

Where Do These Probabilities Come From?

The model’s probabilities are derived from massive corpora. Starting with simple character‑level frequencies, Wolfram shows how to compute unigram, bigram, and higher‑order n‑gram distributions for letters and then for whole words.

He demonstrates that estimating probabilities for all possible word‑pair or longer n‑gram combinations quickly becomes infeasible because the combinatorial space exceeds the amount of text ever written. This limitation motivates the use of neural networks that can generalize to unseen sequences.

ChatGPT’s Internal Structure

ChatGPT is a massive neural network (the current version is a GPT‑3 model with 175 billion parameters) built around the Transformer architecture. The core components are:

Embedding module : converts tokens and their positions into high‑dimensional vectors (768 for GPT‑2, 12 288 for GPT‑3).

Attention blocks : each block contains multiple attention heads (12 in GPT‑2, 96 in GPT‑3) that re‑weight token embeddings based on relevance to other tokens.

Feed‑forward layers : fully‑connected layers that transform the re‑weighted embeddings.

Wolfram includes several diagrams (embedded as tags) that visualize the embedding vectors, attention‑head weight patterns, and the weight matrices of the feed‑forward layers.

After passing through all attention blocks, the Transformer produces a final set of embeddings. The last embedding is decoded into a probability distribution over the next token (about 50 000 possible tokens, of which ~3 000 are whole words).

The entire process is a forward‑only computation; there is no explicit loop inside the network. However, the generation loop exists at the outer level because each new token is fed back as input for the next prediction.

One More Thing

The article itself was edited by ChatGPT, and the model even comments on Wolfram’s original piece.

References: [1] https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ [2] https://twitter.com/stephen_wolfram/status/1625611360967983104 [3] https://writings.stephenwolfram.com/2023/01/wolframalpha-as-the-way-to-bring-computational-knowledge-superpowers-to-chatgpt/