Beginner-Friendly Guide to Understanding Large Language Models

Tokens

A token (词元) is a text fragment that can range from a single character to a whole word. Tokenization (词元化) splits input text into tokens. For the sentence "Hold my math!" :

Word‑level tokenization ["Hold", "my", "math", "!"] Subword‑level tokenization ["Hold", "my", "ma", "th", "!"] Character‑level tokenization

["H", "o", "l", "d", " ", "m", "y", " ", "m", "a", "t", "h", "!"]

In large language models (LLMs) the tokenizer converts the input text into a sequence of tokens before further processing.

Next‑token prediction

LLMs are next‑token predictors: given a sequence of input tokens they output a probability distribution for the next token. Because neural networks accept fixed‑length inputs, they process a limited number of tokens at a time and generate one token per step, iterating to produce longer outputs. An illustration (character‑level tokenization with an input length of five tokens) shows this iterative process.

Token → numeric IDs → embeddings

Neural networks operate on numbers, so each token is mapped to an ID from a predefined vocabulary. Example mappings: the character "H" → ID #8, "m" → ID #13, etc. These IDs are looked up in an embedding matrix, yielding an embedding vector (a fixed‑length numeric array) that represents the token.

Transformer block

The Transformer architecture, introduced in the 2017 paper Attention Is All You Need , underpins modern LLMs. Models such as OpenAI’s GPT‑2 use a decoder‑only stack of Transformer layers, each consisting of a self‑attention sub‑layer and a feed‑forward network (FFN). Residual connections preserve information across layers.

Positional encoding

Positional encoding adds sequence order information to token embeddings so the model can distinguish positions. The positional encoding vectors are added element‑wise to the word embeddings before being fed to the attention sub‑layer. Residual connections ensure this information persists through subsequent layers.

Self‑attention and multi‑head attention

Self‑attention lets the model weigh relationships between tokens. In the example sentence “The cat slept on the mat and it purred,” the token “it” should attend more strongly to “cat” than to other words. The model learns an attention matrix that assigns higher weights to more relevant token pairs. Multi‑head attention combines several such matrices, allowing the model to capture different types of relationships in parallel.

Feed‑forward network and layer normalization

After attention, the output passes through a feed‑forward network, typically an up‑projection followed by a down‑projection. Layer normalization is applied after each sub‑layer (self‑attention and FFN) to stabilize training.

Output linear layer and softmax

The final linear layer produces a vector whose length equals the vocabulary size (e.g., 50 000). Applying the softmax function converts this vector into a probability distribution over all possible next tokens, ensuring the probabilities sum to 1.