Skip to content
Tagged articles

Unigram

2 articles · Page 1 of 1
PaperAgent
PaperAgent
Oct 4, 2026 · Artificial Intelligence

Tokenization: The Hidden Layer Causing 54x AI Cost Gaps Across Languages

Google's 32-author survey on tokenization reveals it as the most underestimated component in LLMs, showing 94% of models reuse identical tokenizers, a 54.3x cost disparity between English and low-resource languages, and that BPE dominates generative models while fundamental fairness and efficiency challenges remain largely unsolved.

BPEHugging FaceSurvey
0 likes · 10 min read
Tokenization: The Hidden Layer Causing 54x AI Cost Gaps Across Languages
AgentGuide
AgentGuide
Apr 12, 2026 · Artificial Intelligence

What Is a Token? A Deep Dive into Tokenization Algorithms for LLMs

The article defines tokens (now officially called “词元”), explains why large language models require numeric input, and details three main tokenization strategies—word‑based, character‑based, and subword—along with the sub‑methods BPE, WordPiece, and Unigram, highlighting their advantages and drawbacks.

BPELLMSubword
0 likes · 6 min read
What Is a Token? A Deep Dive into Tokenization Algorithms for LLMs