LLM Cost Optimization: 8 Engineering Techniques to Reduce API Spend by 92%

Introduction

Many teams building LLM products overlook a critical fact: models charge per token, not per problem difficulty. A startup's customer-service bot answering the same few dozen questions daily racked up a $14,000 OpenAI bill in its second month—not because the product was complex, but because the team never designed for call cost. Even a trivial question sent to a top-tier model incurs the full fee.

This article addresses three core questions: why LLMs burn money faster than expected, why Agent applications amplify the problem, and which cost-reduction techniques to prioritize for different business scenarios.

How LLM Billing Works

Almost all LLM APIs bill by token (roughly 4 English characters per token). Two categories are charged:

Input tokens : everything sent to the model—system prompt, context, user input.

Output tokens : everything the model returns, typically 2–4× more expensive than input tokens.

Approximate 2025–2026 price tiers (per million tokens):

Frontier top-tier : input $15, output $75

Mid-tier : input $3, output $15

Small/Fast : input $0.50, output $1.50

Open-source self-hosted : mainly compute cost

Example: a 2,000-token system prompt sent 10,000 times daily costs 20 million input tokens per day—purely from repeated static content.

Why Agent Applications Are Expensive

Agents loop through multiple model calls: intent classification → tool call → result analysis → response formatting. A single “summarize last month’s sales” query can consume ~10,000 tokens across four steps, costing ~$0.06 on GPT-4o. At 50,000 queries/month that’s ~$3,000 for just one feature. Real agents often burn 30,000–100,000 tokens per task due to memory, context injection, multi-step reasoning, and error recovery. The core issue: every step defaults to an expensive model without caching or judging whether a top-tier model is warranted.

用户请求：“总结上个月的销售表现”

步骤 1：调用 LLM 识别意图         -> 约 800 tokens
步骤 2：调用数据库工具             -> 返回 5000 tokens 数据
步骤 3：调用 LLM 分析数据         -> 6000 tokens 输入，1500 tokens 输出
步骤 4：调用 LLM 格式化响应       -> 2000 tokens 输入，800 tokens 输出

总计：单次查询约 10000 tokens

Five Root Causes of Uncontrolled LLM Costs

Repeated computation of identical prefixes (e.g., same 2,000-token system prompt every request).

Repeated answers to identical FAQs (each triggers full inference).

Using top-tier models for ordinary tasks (not every task needs GPT-5-level capability).

Verbose prompts (a 400-token prompt written as 2,000 tokens).

Uncontrolled output (requesting “detailed analysis” when only three bullet points are needed).

Eight Cost-Reduction Techniques

Technique 1: Prompt Caching (Prefix Caching)

Solves repeated payment for identical prompt prefixes. LLMs compute reusable intermediate states via attention; caching these states avoids recomputation. Anthropic charges ~125% of base input price for cache write, then ~10% for cache read (90% discount), breaking even after ~2 hits. OpenAI offers automatic ~50% discount on repeated prefix tokens.

Maximize hits by ordering prompts: static content first (system prompt, few-shot examples, retrieved documents), dynamic user message last.

错误顺序：
[用户消息]      <- 每次都变
[系统提示]      <- 每次相同

正确顺序：
[系统提示]      <- 静态，可缓存
[Few-shot 示例] <- 静态，可缓存
[检索文档]      <- 会话内相对稳定，可缓存
[用户消息]      <- 动态，放最后

A RAG app attaching 3,000-token documents per request can cut input cost 80–90% via prefix caching alone.

Technique 2: Semantic Caching

Solves re-inference for semantically identical questions phrased differently. Instead of string matching, embed requests as vectors, retrieve similar historical queries from a vector store, and return cached answers if cosine similarity exceeds a threshold (e.g., 0.93). “How to reset password?” and “I forgot my password” hit the same cache.

Production data: customer-service apps see 40–60% hit rates; FAQ bots 70–80%. Redis LangCache reported 73% cost reduction under high-repeat loads. Cache latency is milliseconds vs. seconds for re-inference.

Starter tools: GPTCache (open-source, LangChain integration), Redis + Vector Search (production-grade concurrency), pgvector (natural if already on Postgres). Calibrate threshold carefully: too low returns wrong answers; too high loses savings. Start at 0.93.

Technique 3: Reduce Token Usage

Output tokens are often pricier, so controlling output length directly controls cost. Three practical tactics:

Enforce strict output formats —e.g., require JSON with fixed fields instead of free-form prose.

# 可能花掉约 800 个输出 Token：
"Analyze this customer feedback and share your thoughts."

# 可能只需约 60 个输出 Token：
"""
Analyze this feedback.
Respond ONLY with:
{
  "sentiment": "positive/negative/neutral",
  "key_issue": "one sentence max",
  "action_needed": true/false
}
"""