Why Large Language Models Measure Usage in Tokens

What is a Token?

Token is the smallest computational unit that a large language model uses to process text. It can be a word, character, subword, or punctuation. The model does not treat the whole sentence as a whole; it first splits the input into tokens.

Example: the Chinese sentence “我爱人工智能。” is tokenized into ["我","爱","人工","智能","。"]. Each token is the unit the model processes, similar to how a byte is fundamental to a computer.

Why Models Use Tokens for Measurement

Running large models is computationally expensive. Token processing involves GPU compute, memory bandwidth, and API transmission, which together account for most of the inference cost (GPU 68%, memory 22%, transmission 7%, system overhead 3%). Charging per token therefore provides a precise, fair, and transparent way to reflect resource usage.

Each token’s generation requires substantial computation, storage, and network transmission, so pricing by token aligns cost with actual resource consumption.

Underlying Logic of Tokens

Tokens are generated by a tokenizer. The dominant method is Byte Pair Encoding (BPE), which starts from individual characters and merges frequent character sequences into subword tokens such as “love”. This reduces token count while preserving language structure.

Characteristics of Tokens

Not a word nor a character : a token can be a letter, Chinese character, word, or punctuation. Example: “you” is one token, “你” is also one token.

Variable length : some words become multiple tokens, e.g., “Artificial” → ["Art","ifi","cial"].

Cross‑language : the same tokenizer works for English, Chinese, Japanese, etc.; OpenAI’s tokenizer supports almost all languages.

Billable unit : usage and pricing are calculated per token (e.g., 1 000 tokens ≈ 750 English words ≈ 500 Chinese characters).

How Tokens Affect Applications

Document Q&A : overly long token sequences may exceed model limits and need chunking.

Multi‑turn dialogue : each turn adds tokens, making longer conversations more costly.

Cost control : providers price per token (e.g., OpenAI $0.01 per 1 K tokens).

Model training : models are trained on trillions of tokens.

Tokens in Conversation and Text Generation

During a chat, every word and punctuation is tokenized; the model understands and predicts token by token, then generates responses token by token.

Example: the query “请介绍一下AI的发展。” becomes ["请","介绍","一下","A","I","的","发展","。"] and the model processes these tokens sequentially to produce a reply.

Tokens are the foundational units that enable large language models to convert complex language into manipulable numeric sequences, powering dialogue, writing, and question‑answering capabilities.