Demystifying LLMs: How Tokens, Training, and Transformers Power Generative AI

Why LLMs Appear Intelligent

Generative AI is everywhere, and many people wonder how large language models (LLMs) actually work. In reality, LLMs do not understand language; they simply receive a text prompt and predict the next token (the next word or sub‑word unit).

Tokens and Vocabulary

A token is the basic unit an LLM processes. Tokens can be whole words, sub‑words, punctuation, or spaces. LLMs use a vocabulary of tokens, typically built with a Byte‑Pair Encoding (BPE) algorithm. For example, the open‑source GPT‑2 model has a vocabulary of 50,257 tokens.

Python developers can explore tokens with the tiktoken package:

pip install tiktoken

import tiktoken
encoding = tiktoken.encoding_for_model("gpt-2")
print(encoding.encode("The quick brown fox jumps over the lazy dog."))
print(encoding.decode([464, 2068, 7586, 21831, 18045, 625, 262, 16931, 3290, 13]))

In this example token 464 corresponds to "The", 2068 to " quick" (including a leading space), and 13 to the period.

Predicting the Next Token

Given a sequence of tokens, an LLM predicts a probability distribution over the entire vocabulary for the next token. A simple Python‑style pseudocode illustrates the idea:

def get_token_predictions(input_tokens):
    last_token = input_tokens[-1]
    return probabilities_table[last_token]

In practice the probability table would be replaced by a neural network that computes these probabilities on the fly.

Simple Probability Table Example

Consider a tiny vocabulary ['I', 'you', 'like', 'apples', 'bananas'] and three training sentences:

I like apples

I like bananas

you like bananas

Counting token pairs yields the following probabilities (shown here in prose):

"I" is always followed by "like" (100%).

"you" is always followed by "like" (100%).

"like" is followed by "apples" 33.3% of the time and by "bananas" 66.7% of the time.

"apples" and "bananas" have no observed successors; we can assign a uniform fallback distribution to the other tokens.

This tiny model demonstrates how a Markov chain can generate text by repeatedly sampling the next token based on the current one.

Context Window Limitations

A Markov chain that looks at only the last token has a context window of size 1, which leads to incoherent output. Extending the window to two or three tokens improves coherence but still falls short of human‑like reasoning. Modern LLMs use much larger windows: GPT‑2 uses 1024 tokens, GPT‑3 up to 2048, GPT‑4 up to 8192 (and later versions even larger).

From Markov Chains to Neural Networks

Storing probability tables for large windows is infeasible, so LLMs replace tables with neural networks. A neural network receives token IDs as input and outputs a probability distribution for the next token. Training adjusts billions of parameters (e.g., GPT‑2 ≈ 1.5 billion, GPT‑3 ≈ 175 billion, GPT‑4 ≈ 1.76 trillion) using back‑propagation on massive text corpora.

Transformers and Attention

The dominant architecture for LLMs is the Transformer, which relies on an attention mechanism to relate every token in the context window to every other token. This allows the model to capture long‑range dependencies and produce coherent continuations.

Are LLMs Truly Intelligent?

LLMs do not reason or generate original ideas; they stitch together patterns learned from training data. Their outputs can appear creative, but they may also hallucinate facts. Consequently, any LLM‑generated content should be verified by humans before being presented to end users.

Author: Architect