How Much Can AI Remember? Understanding Tokens and Context Windows

Large language models do not work with characters or words directly; they first split input text into numeric units called tokens . A token can be a whole word, a sub‑word, a part of a Chinese character, or a frequent phrase, depending on the model’s vocabulary and the tokenization algorithm.

Tokenization and Chinese Text

Modern models typically use Byte‑Pair Encoding (BPE). The algorithm merges frequently co‑occurring byte pairs into larger chunks, so high‑frequency sequences become single tokens while rare characters remain fragmented. Because a Chinese character occupies three UTF‑8 bytes, Chinese text often yields more tokens than English. Statistics show Chinese averages 1.5–2.5 tokens per character, whereas English averages about 1.3 tokens per word, meaning Chinese can consume roughly 1.5× the tokens of English for the same meaning.

Examples: the common word "中国" is stored as one token, while the rare character "夔" is split into three tokens. Consequently, the same sentence can cost significantly more tokens—and therefore more money—when processed by a model that has not been optimized for Chinese.

Context Window: The Model’s Short‑Term Memory

The context window defines the maximum number of tokens a model can attend to in a single request, including the user prompt, system messages, prior conversation history, tool results, and the model’s own generated output. Early models had a 4 K token window; later versions expanded to 8 K, 32 K, 200 K (Claude), and even 2 M tokens (Gemini).

Even the largest 2 M‑token window cannot hold a 1‑million‑character Chinese novel, which would require 1.5–2.5 M tokens. When the input exceeds the window, the excess is truncated or the request is rejected.

Engineering Strategies for Long Texts

To process documents that exceed the window, engineers employ three main techniques:

Sliding window : Keep the most recent conversation turns and compress earlier parts into a summary, preserving recent details while limiting token growth.

Map‑Reduce (divide‑and‑conquer) : Split the document into chunks, run the model on each chunk independently (Map), then aggregate the partial results into a final answer (Reduce). This enables parallel processing of very long texts.

Recursive summarization / context compression : Summarize each chunk, then re‑summarize the summaries iteratively. Reported compression rates can reduce a 100 k‑token technical document to 1 k tokens, achieving over 90 % reduction.

Token Cost Implications

Tokens are also the billing unit for most LLM services. For example, GPT‑4 Turbo charges roughly $0.01 per 1 000 input tokens and $0.03 per 1 000 output tokens, while some domestic models charge about 0.8 CNY per million input tokens. Because Chinese text consumes more tokens, the same content can cost up to twice as much.

Prompt engineering matters: a 500‑character Chinese paragraph may contain ~750 tokens. Removing filler or redundant phrasing can save hundreds of tokens, reducing both latency and cost.

Conclusion

The apparent inability of AI to read an entire novel stems from the physical limit of the context window measured in tokens, not from a lack of intelligence. Understanding tokenization, context‑window size, and cost dynamics, and applying engineering tricks such as sliding windows, map‑reduce, and recursive summarization, allow practitioners to handle long texts efficiently and economically.