How LLMs Turn Text into Numbers: Tokenization & Embeddings Explained

1. Dictionary

Natural language cannot be fed directly into neural networks like image pixels; it must be turned into numbers that preserve semantic relationships. The first step is a dictionary (vocabulary) that maps text to tokens via tokenization.

Using DeepSeek-V3.1-Base as an example, the tokenizer files are downloaded with hfd.sh and loaded via Hugging Face Transformers:

from transformers import AutoTokenizer

text = "一个高尚的人，一个纯粹的人，一个有道德的人，一个脱离了低级趣味的人，一个有益于人民的人。"
tokenizer = AutoTokenizer.from_pretrained("./deepseek-tokenizer")

vocab_size = tokenizer.vocab_size
print(f"字典大小: {vocab_size}个token
")

tokens = tokenizer.encode(text)
print(f"原始文本: {text}")
print(f"Token IDs: {tokens}")
print(f"Token数量: {len(tokens)}
")

print("Token分析:")
for i, token_id in enumerate(tokens):
    token_text = tokenizer.decode([token_id])
    print(f"位置{i}: ID={token_id}, 内容='{token_text}'")

Output:

字典大小: 128000个token

原始文本: 一个高尚的人，一个纯粹的人，一个有道德的人，一个脱离了低级趣味的人，一个有益于人民的人。
Token IDs: [0, 1057, 61679, 2264, 303, 1057, 38304, 2264, 303, 115795, 12247, 2264, 303, 1057, 124256, 102507, 45052, 2264, 303, 1057, 40481, 766, 3641, 2264, 320]
Token数量: 25

Token分析:
位置0: ID=0, 内容='<｜begin▁of▁sentence｜>'
位置1: ID=1057, 内容='一个'
位置2: ID=61679, 内容='高尚'
位置3: ID=2264, 内容='的人'
位置4: ID=303, 内容='，'
位置5: ID=1057, 内容='一个'
位置6: ID=38304, 内容='纯粹'
位置7: ID=2264, 内容='的人'
位置8: ID=303, 内容='，'
位置9: ID=115795, 内容='一个有'
位置10: ID=12247, 内容='道德'
位置11: ID=2264, 内容='的人'
位置12: ID=303, 内容='，'
位置13: ID=1057, 内容='一个'
位置14: ID=124256, 内容='脱离了'
位置15: ID=102507, 内容='低级'
位置16: ID=45052, 内容='趣味'
位置17: ID=2264, 内容='的人'
位置18: ID=303, 内容='，'
位置19: ID=1057, 内容='一个'
位置20: ID=40481, 内容='有益'
位置21: ID=766, 内容='于'
位置22: ID=3641, 内容='人民'
位置23: ID=2264, 内容='的人'
位置24: ID=320, 内容='。'

Key observations:

Fixed vocabulary size : 128,000 tokens in this example.

Special tokens : e.g., <｜begin▁of▁sentence｜> marks sequence start. Others include <｜end▁of▁sentence｜> (end of generation) and <｜▁pad▁｜> (padding for batching), listed in tokenizer.json under added_tokens.

Longest-match-first tokenization : "一个有" (one token) is preferred over "一个" + "有" (two tokens).

How is the dictionary built? The tokenizer.json reveals BPE (Byte Pair Encoding) :

{
  "model": {
    "type": "BPE"
  }
}

BPE core idea: iteratively merge the most frequent adjacent character/token pairs until the target vocabulary size is reached.

Initialize vocabulary : split text into characters; each character is a token.

Count pair frequencies : under current vocabulary, segment text into tokens and count adjacent token-pair frequencies.

Merge most frequent pair : combine the top pair into a new token, add to vocabulary.

Repeat : steps 3–4 until vocabulary reaches preset size.

Token IDs alone are insufficient for neural networks because they are arbitrary indices — numerical distance does not reflect semantic similarity, and the same token can have different meanings in different contexts (e.g., "苹果" as fruit vs. brand).

2. Word Embeddings

Word embeddings solve this by mapping tokens to low-dimensional dense vectors where semantically similar tokens have nearby vectors. The seminal project is Google's Word2Vec (CBOW and Skip-gram), both shallow neural networks. Though modern LLMs use different training objectives, the embedding concept remains the same.

CBOW predicts the center word from context.

Input: one-hot vectors of surrounding words (e.g., "今天", "很好")

Hidden layer: average of surrounding word vectors

Output: probability distribution over vocabulary for center word (e.g., "天气")

Training: softmax loss, backpropagation updates word vectors

Skip-gram predicts context from the center word.

Input: one-hot vector of center word (e.g., "天气")

Hidden layer: directly retrieve the center word's embedding

Output: probability distribution for each surrounding word (e.g., "今天", "很好")

Training: compute loss per surrounding word and update

The network is shallow but wide: input and output layer sizes equal vocabulary size; embedding dimension is determined by hidden layer size.

To see embeddings in a real LLM, the author loads DeepSeek-LLM-7B-Chat (since V3.1 is too large):

from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn.functional as F

model_name = "./deepseek-llm-7b-chat"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
model.eval()

embedding_layer = model.get_input_embeddings()

print("=== Embedding层信息 ===")
print(f"Embedding权重形状: {embedding_layer.weight.shape}")  # [vocab_size, hidden_size]
print(f"词汇表大小: {embedding_layer.weight.shape[0]}")
print(f"隐藏层维度: {embedding_layer.weight.shape[1]}")

# Inspect a specific token's embedding
hello_id = tokenizer.encode("hello")[1]  # get "hello" token ID
hello_embedding = embedding_layer.weight[hello_id]
print(f"\"hello\"的embedding维度: {hello_embedding.shape}
")

def calculate_similarity(word1, word2):
    """计算两个词语的cosine相似度"""
    word1_id = tokenizer.encode(word1)[1]
    word2_id = tokenizer.encode(word2)[1]
    embedding_layer = model.get_input_embeddings()
    with torch.no_grad():
        emb1 = embedding_layer.weight[word1_id]
        emb2 = embedding_layer.weight[word2_id]
    similarity = F.cosine_similarity(emb1.unsqueeze(0), emb2.unsqueeze(0))
    print(f"'{word1}' 和 '{word2}' 的相似度: {similarity.item():.4f}")
    return similarity

# Test similarity
calculate_similarity("cat", "dog")
calculate_similarity("apple", "banana")
calculate_similarity("computer", "river")

Output:

=== Embedding层信息 ===
Embedding权重形状: torch.Size([102400, 4096])
词汇表大小: 102400
隐藏层维度: 4096
"hello"的embedding维度: torch.Size([4096])

'cat' 和 'dog' 的相似度: 0.1846
'apple' 和 'banana' 的相似度: 0.0050
'computer' 和 'river' 的相似度: 0.0615

The 7B model has a vocabulary of 102,400 and embedding dimension 4,096. Cosine similarity shows "cat" and "dog" are more similar (0.1846) than "apple" and "banana" (0.0050) or "computer" and "river" (0.0615).

Because embeddings capture similarity, they enable probability computation over the vocabulary, which is the basis for next-token prediction — the core logic behind ChatGPT.