What Is a Token? The Key to Understanding AI’s Billing Unit

1. What a Token Is

Many assume a token equals a Chinese character or an English word, but a token is actually the smallest unit a model processes, defined by the tokenizer and sitting between characters and words.

English Tokenization

Original: "understanding"
Tokens: ["under", "stand", "ing"] → 3 tokens

Original: "cat"
Tokens: ["cat"] → 1 token

Original: "Hello, world!"
Tokens: ["Hello", ",", " world", "!"] → 4 tokens

Common English words are usually one token; longer or rare words are split.

Chinese Tokenization

Original: "你好世界"
Tokens: approx. 6~8 tokens (varies by model)

Original: "Hello"
Tokens: 1 token

Conclusion: The same meaning consumes more tokens in Chinese than in English.

Rough Conversion Rules

English: 1000 tokens ≈ 750 words
Chinese: 1000 tokens ≈ 500 characters
Code: 1000 tokens ≈ 600~700 characters (depends on language)

2. Why Tokens Matter

Cost : API charges are based on input + output tokens.

Speed : More generated tokens mean slower responses.

Context : The amount of text a model can “see” at once is limited by the token window.

Quality : When the token window is insufficient, the model forgets earlier content.

3. Context Window Explained

The context window is the maximum amount of text, measured in tokens, that a model can consider before generating a response.

It can be visualized as the model’s “desktop” where previous dialogue, system prompts, uploaded files, and the current input all occupy space.

┌──────────────────────────────────────────┐
│          Model’s Desktop (Context Window)│
│                                          │
│  Your earlier statements                  │
│  ─────────────────────────────────────── │
│  Model’s previous replies                │
│  ─────────────────────────────────────── │
│  Uploaded files / pasted code             │
│  ─────────────────────────────────────── │
│  System prompt                           │
│  ─────────────────────────────────────── │
│  ← Current input                         │
└──────────────────────────────────────────┘
Total tokens cannot exceed the window limit.

Typical Context Windows

GPT‑4o: 128K tokens

Claude 3.7 Sonnet: 200K tokens

Gemini 1.5 Pro: 1M tokens

DeepSeek V3: 128K tokens

128K tokens can hold roughly a 100‑k‑character Chinese novel, but in agent scenarios the combined tool descriptions, history, and retrieval results can quickly approach the limit.

4. What Happens When You Exceed the Window?

Two outcomes are possible:

Error : Most APIs return an error indicating the token limit was exceeded.

Forgetting : Some applications truncate the earliest dialogue, keeping only recent turns, which leads to the model “forgetting” earlier statements.

Conversation history (top‑to‑bottom):

Round 1: You said "I am Xiao Ming" ← truncated
Round 2: Discussed project background ← truncated
…
Round 20: You said "Write an email, sign with my name"
Model: "Sure, what is your name?" ← it no longer remembers.

This explains why long conversations sometimes appear to lose context—not because the model degrades, but because the window is full.

5. Input vs. Output Tokens

API calls charge separately for input and output tokens.

# Pseudocode: token composition of a single API call
response = llm.chat(
    messages=[
        {"role": "system", "content": "You are an assistant"},   # input token
        {"role": "user", "content": "Write a poem"},           # input token
    ]
)

# response.usage reports:
{
    "input_tokens": 15,   # tokens you sent (including system prompt)
    "output_tokens": 80,  # tokens generated by the model
    "total_tokens": 95
}

# Cost = input_tokens × input_rate + output_tokens × output_rate
# Typically, output token price is 3–5× the input price.

Output tokens are more expensive because generation requires sequential computation for each token.

6. Why Longer Outputs Are Slower

The model predicts one token at a time, adds it to the context, then predicts the next token.

Generation (simplified):

Input: "Today's weather"
Step 1: predict next token → "very" → context becomes "Today's weather very"
Step 2: predict next token → "nice" → context becomes "Today's weather very nice"
Step 3: predict next token → "," → context becomes "Today's weather very nice,"
…
Generating 100 tokens = 100 loops
Generating 2000 tokens = 2000 loops

Thus, longer outputs increase latency and cost, which is why concise answers are faster and cheaper.

7. Tokens and Context Engineering

Complex AI applications (agents) must fit system prompts, dialogue history, retrieved documents, and tool descriptions into a limited token window.

Context Engineering studies how to decide what to include, how to arrange it, and what to omit.

Practical principles:

Principle 1: Include Only Necessary Content

# ❌ Add all history
context = all_history + current_question

# ✅ Keep recent rounds + key summary
recent = history[-6:]               # last 6 turns
summary = summarize(history[:-6])   # compress older part
context = [summary_msg] + recent + [current_question]

Principle 2: Put Important Content at the Beginning and End Models attend more to the start and end of the context (the “Lost in the Middle” effect).

# ✅ Place most relevant documents first, less relevant later
context = [most_relevant, secondary1, secondary2, backup_doc, user_question]

Principle 3: Load Tool Descriptions on Demand

# ❌ Load descriptions of all 100 tools (wastes tokens)
tools_in_context = all_100_tools

# ✅ Load only tools relevant to the user’s intent
intent = classify(user_input)   # e.g., email / search / calendar
tools_in_context = relevant_tools[intent]  # 5–10 tools

8. Quick FAQ (One‑Minute Token Cheat Sheet)

Q: How many words/characters are 1000 tokens? A: About 750 English words or 500 Chinese characters.

Q: Why does the same query cost more today than yesterday? A: Today’s context likely includes more dialogue history, increasing input tokens.

Q: Why does the AI sometimes “forget” what I said earlier? A: The conversation exceeded the context window, and early content was truncated.

Q: How can I save tokens? A: Ask for concise answers, prune unnecessary history, and keep system prompts brief.

Q: Is a larger context window always better? A: Not necessarily. An overly full window can dilute attention and hurt quality; precision beats sheer size.

Conclusion

Tokens are the fundamental “currency” of AI models; they drive cost, speed, and capability limits. Mastering token usage enables you to control expenses, optimize performance, and design effective context‑management strategies, ultimately raising the quality ceiling of your AI products.