How Long Does a Million LLM Tokens Last and How to Cut the Cost?

1. What Is a Token?

A token is the basic unit a model processes after tokenization. It is not equivalent to characters or words. Examples include a single Chinese character, a common English word, a sub‑word, a number/punctuation/space combination, or a code keyword/identifier. Consequently, token count varies with language, rare characters, mixed scripts, long numbers, URLs, JSON, Base64, and code.

2. Where Does the Money Go?

Most LLM services split consumption into three parts:

Regular input tokens : content the model must re‑process.

Cached input tokens : repeated content that hits the prompt cache.

Output tokens : the model’s generated answer and any internal reasoning tokens.

A generic cost formula is:

Cost = regular_input_tokens × regular_input_price
     + cached_input_tokens × cache_read_price
     + output_tokens × output_price
     + other_possible_fees

Additional fees may include cache‑write costs, image input, audio/video processing, web search, tool calls, or batch‑processing discounts, depending on the platform.

3. Why Does One Simple Question Consume Many Input Tokens?

LLM services do not retain conversation state internally. Each request typically resends the system prompt, full conversation history, tool definitions, and any project files. For example, a three‑turn dialogue might be sent as:

Turn 1: system_prompt + user_question_1
Turn 2: system_prompt + question_1 + answer_1 + user_question_2
Turn 3: system_prompt + question_1 + answer_1 + question_2 + answer_2 + user_question_3

If each turn adds ~2,000 tokens, the cumulative input after three turns is 12,000 tokens, not 6,000. In AI‑coding scenarios the context can also contain repository rules, open files, search results, diffs, compiler output, tool parameters, and previous model analyses, quickly reaching tens of thousands of tokens per turn.

4. Why Prompt Cache Is Cheap but Not Free

Prompt cache stores the pre‑processed representation of a long, stable prefix the first time it is read. Subsequent requests that reuse the same prefix can skip the heavy computation, lowering both compute cost and latency. However, cache reads still consume storage, VRAM, bandwidth, and scheduling resources, so they are billed at a reduced rate. Some platforms also charge a one‑time cache‑write fee.

Cache hit depends on:

Exact content match

Identical ordering

Cacheable part being at the prompt prefix

Meeting the provider’s minimum cache length

Cache still being within its validity period

No change in model, region, or routing

Even a tiny change (e.g., inserting a timestamp at the start of a long system prompt) can invalidate large portions of the cache.

5. Why Output Tokens Are Usually More Expensive

Generating each new token requires the model to recompute based on all prior context, so output tokens consume more compute than input tokens and therefore have a higher unit price.

For models that support deep reasoning, output tokens are split into:

Visible output: the answer shown to the user
Reasoning tokens: internal computation before the final answer

Different products expose reasoning tokens differently, but they still affect usage and cost. Controlling output length and reasoning intensity is often more effective than repeatedly trimming input.

6. How Long Can One Million Tokens Actually Last?

The usable number of interaction rounds can be approximated by:

Rounds ≈ token_quota ÷ average_tokens_per_round

If a lightweight Q&A consumes ~2,000 tokens per round, a million‑token quota supports roughly 500 rounds. Longer contexts reduce the number of rounds. Illustrative scenarios:

Short Q&A / translation / polishing: 500–2,000 tokens per round → hundreds to a few thousand rounds.

Long document summarization / data analysis: 5,000–30,000 tokens per round → tens to a couple of hundred rounds.

Multi‑file code analysis: 20,000–100,000 tokens per round → about 10–50 rounds.

Large‑scale agent or long‑running automation: >100,000 tokens per round → only a few rounds.

In AI‑coding tools, the rapid consumption is not only due to code size but also because agents repeatedly fetch files, run commands, capture logs, and feed those results back into the model.

7. Converting Tokens to Actual Cost

Assume a model’s pricing is:

Regular input: 2 CNY / 1M tokens
Cache input:   0.4 CNY / 1M tokens
Output:        8 CNY / 1M tokens

For a request with the following usage:

Regular input: 100,000 tokens
Cache input:   400,000 tokens
Output:        50,000 tokens

The cost is calculated as:

100,000 ÷ 1,000,000 × 2 = 0.20
400,000 ÷ 1,000,000 × 0.4 = 0.16
50,000 ÷ 1,000,000 × 8 = 0.40
Total = 0.76 CNY

This example shows that looking only at total token count is misleading; the most expensive part can be the output even when cache input dominates the volume.

8. Effective Token‑Saving Strategies

1. Trim Long, Automatically Injected Rule Files

Project descriptions, system rules, and agent configurations are often loaded every round. Remove any information that can be inferred from code or is rarely needed.

2. Compress Context Instead of Frequently Clearing It

Rather than discarding the entire history, periodically summarize confirmed requirements, conclusions, and next steps into a short snippet, then start a new conversation with that summary.

3. Control Terminal Logs and Tool Output

Tool results (e.g., build logs, full JSON dumps, dependency trees) can quickly fill the context. Keep only the essential lines, search‑then‑read file fragments, limit test scope, and extract only needed fields from large JSON objects.

4. Adjust Reasoning Strength by Task

Use stronger models for architecture design, complex debugging, or cross‑file refactoring, and cheaper models for simple rewrites, title generation, or straightforward explanations.

5. Explicitly Request Output Length and Format

Prompt the model with constraints such as “summarize in three sentences” or “output no more than 300 characters” to avoid unnecessary token generation.

6. Delegate Tasks to Appropriate Models

Assign classification, extraction, or simple formatting to lightweight models, reserving the most capable model for high‑risk decisions and deep reasoning.

7. Ensure Prompt Cache Hits

Place stable content (rules, examples, tool definitions) at the beginning of the prompt and keep dynamic parts (timestamps, request IDs, user queries) at the end. Avoid reordering stable prefixes.

9. Why Compressing Tool Output Must Be Done Carefully

Compression is inherently lossy. Dropping details in code or error logs can remove the very information needed for debugging. Moreover, reducing tool‑output tokens does not directly lower the cost of the model’s final answer or internal reasoning.

10. Common Misconceptions

1. One Million Tokens Equals One Million Characters

Tokenization differs across models; the same text can yield different token counts.

2. Only the User’s Question Counts

Actual input also includes system prompts, conversation history, tool definitions, and file contents.

3. Cache Hits Are Free

Cache reads are billed at a reduced rate, and some providers also charge for cache writes.

4. Clearing Context Is Always Cheapest

Clearing forces the model to re‑read project information, which may increase overall cost.

5. Unit Price Alone Determines Cost

A cheaper model that requires more rounds or generates more errors can end up more expensive.

11. Metrics Development Teams Should Monitor

Per‑request breakdown of regular input, cache input, output, and reasoning tokens.

Cache‑hit rate.

Average number of rounds per task.

Success rate and average cost per model variant.

Unit cost per user, feature, or workflow.

P50/P95 request cost and latency.

Waste caused by context truncation, retries, and failed calls.

More valuable than raw token counts is the cost per successful task, such as average model cost to fix a bug or generate a report.

Conclusion

LLM billing can be reduced to three questions: how many tokens were read, how many hit the cache, and how many were generated. Regular input determines how much new information the model must understand; cache input saves computation; output and reasoning tokens usually dominate the bill. Effective optimization combines clean context, stable caching, focused tool output, appropriate reasoning strength, and task‑level model delegation.