Machine Learning Algorithms & Natural Language Processing

Oct 9, 2026 · Artificial Intelligence

Why LLMs Can't Count Letters in 'Strawberry': The Tokenization Problem Explained

This article explains how tokenization works in large language models, why it causes failures like miscounting letters in 'strawberry', details the BPE algorithm, discusses costs like language inequality and security risks, and explores alternatives like byte-level and dynamic tokenization.