How Tokenizers and Embeddings Encode Language – The Mechanics Behind LLMs

Why Tokenization Matters

A developer observed that sending the same technical document to GPT‑4 in Chinese consumed 1.77 × more tokens than in English, raising API costs by the same factor. Petrov et al. (NeurIPS 2023) measured that token counts can differ by up to 15 × across languages, meaning non‑English users may pay an order of magnitude more for the same semantic content.

Token vs. Word vs. Character

Splitting text by whitespace (word tokenization) fails for out‑of‑vocabulary words, morphological variations, languages without spaces, and code or numbers. Character‑level tokenization avoids OOV issues but produces excessively long sequences, reducing usable context length. Subword tokenization balances the two by keeping frequent words intact and breaking rare words into meaningful fragments (e.g., "running" → ["run", "ning"]).

One‑sentence summary: pure word tokenization leads to endless OOVs, spelling variations, space‑less languages, and code/number problems; the vocabulary can never be large enough, and generalisation is always insufficient.

Subword Algorithms

BPE (Byte Pair Encoding) was introduced by Gage (1994) as a compression algorithm and adapted for neural machine translation by Sennrich et al. (ACL 2016). It iteratively merges the most frequent adjacent character pairs until a target vocabulary size is reached.

WordPiece (Schuster & Nakajima, ICASSP 2012) selects merges that maximise the likelihood of the training corpus and was used in BERT (Devlin et al., NAACL 2019) with a 30 k vocabulary.

Unigram Language Model (Kudo, ACL 2018) starts from a large candidate list and deletes tokens that contribute least to corpus likelihood, supporting multiple segmentation candidates (subword regularisation).

SentencePiece (Kudo & Richardson, EMNLP 2018) provides a language‑agnostic interface that can train BPE or Unigram models directly from raw text, eliminating the need for pre‑tokenisation.

Byte‑level BPE (Radford et al., OpenAI 2019) operates on byte sequences, yielding a 256‑byte base vocabulary that removes UNK completely; GPT‑2 uses a 50 k vocabulary, while GPT‑3/4 use ~100 k.

Technical note: the industry claim "GPT‑4 supports 100+ languages" is technically accurate for exposure to multilingual data, but the tokenizer’s vocabulary design can cause up to 15 × efficiency differences, so "support" does not equal "equal efficiency".

Special Tokens and Chat Templates

Beyond subwords, tokenizers include control tokens such as BOS, EOS, PAD, UNK, CLS, SEP, MASK, and role tokens (e.g., <|im_start|>user, [INST]...[/INST]). These define sequence boundaries and speaker roles. Using the wrong chat template (format) degrades model performance, as shown by HuggingFace (2023) where Llama‑2 loses ~7 % accuracy without the proper template.

Engineers often confuse Chat Template (format) with System Prompt (content); the former affects the model’s basic capability, the latter influences response quality.

Embedding and Positional Encoding

Token IDs are looked up in an embedding matrix of size V × d (e.g., V = 128 256, d = 4096 → 5.25 × 10⁸ parameters, ~7.5 % of a 7 B model). Positional information is added via sinusoidal encodings (Vaswani et al., NeurIPS 2017) or RoPE (used in Llama series).

Weight tying (Press & Wolf, EACL 2017; Inan et al., ICLR 2017) shares the input embedding matrix with the output projection, reducing parameters without hurting performance.

Multilingual Token Costs

Using GPT‑4’s cl100k_base, 100 English words ≈ 130 tokens, the same Chinese text ≈ 300 tokens, Japanese ≈ 250, Hindi ≈ 350. Llama 3 expanded its vocabulary from 32 k to 128 k to improve Chinese token efficiency.

Engineering Evaluation

Good tokenizers balance compression rate, fertility (average tokens per word), UNK rate, and multilingual fairness. Before continued pre‑training, measure fertility on target corpora; if Chinese fertility > 2.0, expanding the vocabulary may be worthwhile.

Extending a vocabulary introduces risk: new tokens start with random or averaged embeddings and may under‑fit if insufficient fine‑tuning data is available.

Practical Guidelines

Do not treat the tokenizer as a trivial preprocessing step; it directly impacts context capacity, training FLOPs, inference speed, and API cost.

Keep the same tokenizer across embedding and generation models to avoid retrieval quality loss.

Remember that token embeddings differ from text‑level embeddings used for vector search.

Capabilities and Limits

A tokenizer enables language‑agnostic conversion of text to token IDs, eliminating UNK and allowing arbitrary input. It does not raise the model’s upper‑bound intelligence, but a poor tokenizer can lower the lower‑bound performance for specific languages or tasks.

Failure example: a Chinese application built on Llama‑2 (32 k vocab) suffered high token fertility (~2.5), leading to reduced context, slower inference, and higher cost; switching to a 128 k vocabulary restored efficiency.

Conclusion

Tokenizers split text into subwords; embeddings map those IDs to continuous vectors. Together they determine how a model "sees" language, influencing context length, training cost, and inference expense.

Core data structure: Token ID → Embedding Matrix (V × d) lookup → Token Embedding + Positional Embedding → model input.

Next article: the self‑attention mechanism that lets each token attend to every other token, replacing RNNs.