Agent Cost Optimization: Cut Waste, Not Intelligence

One: The First Hidden Waste — Re‑understanding the Same Thing Every Turn

Before generating a response, a model processes the system prompt, tool definitions, long‑term rules, context, and conversation history to build an internal state (the prefill phase). Prompt caching saves the KV cache so that if the next request shares the same prefix, the system reuses that state instead of recomputing. However, “same” means byte‑level identity: same model, identical cached content, and within the TTL (default 5 minutes).

Common cache‑breakers include changing effort or thinking settings mid‑conversation, injecting dynamic values (timestamps, random IDs) at the start of the system prompt, reordering tool definitions, forking sessions across models or effort levels, and tool/sub‑task execution exceeding the TTL.

The portable principle: “Put the most stable, expensive, frequently reused content first; put growing, changing content last.” Static rules, tool schemas, project specs, and knowledge packs should form a stable prefix; user input, fresh tool results, and transient state belong at the end.

Practical strategies from the tutorial:

Monitor hit rate, don’t guess. Use cache diagnostics to locate the first divergence point when hits drop.

Defer rarely used tools. Mark them defer_loading so they enter the session only when the model actually queries them.

Add runtime rules as session messages. Append a message instead of rewriting the system prompt.

Switch config only when the cache is already invalid. Context compression rewrites history; that’s the right moment to change model or effort.

Let the breakpoint move forward with the conversation. Auto‑caching can place the breakpoint on the last cacheable block.

Pre‑warm the cache. Send a request with max_tokens: 0 and an explicit breakpoint while the user is still typing.

Mind the TTL. If a parent agent waits too long for a tool or sub‑task, the cache may expire; evaluate whether a 1‑hour TTL is worth it.

This “static prefix + dynamic tail” pattern applies to any model API (OpenAI Responses, Gemini, open‑source serving) or custom agent runner.

Two: The Second Waste — Carrying Old Model “Guardrails” into New Models

Prompts accumulate like legacy wiring: every failure adds a rule, and no one knows which are still needed. The tutorial identifies six prompting anti‑patterns :

Verification rituals — e.g., “double‑check before answering.”

Over‑emphasis — e.g., “be extremely thorough,” “CRITICAL: MUST ALWAYS…”

Forced procedures & hand‑written scratchpads — requiring fixed steps or writing every thought to a scratchpad.

Stale few‑shot examples — crafted for an older model, now inducing verbose reasoning.

Contradictory rules .

Obsolete configs — e.g., hand‑written thinking budgets no longer supported.

These look like “safer, more careful” instructions, but frontier models execute them literally. A customer‑support benchmark compared a base prompt against six legacy‑prompt variants (old thinking settings, contradictory refund rules, hand‑written scratchpad, “verify twice,” “be maximally thorough,” forced six‑step procedure). After cleaning anti‑patterns, cost dropped 14.6% while accuracy rose 5.3% .

Concrete failure modes:

“Verify twice” duplicated order queries for every refund.

“Be maximally thorough” expanded a single support case into dozens of unnecessary knowledge‑base searches.

Hand‑written scratchpad collided with the model’s built‑in reasoning; the model wrote tool calls into the reasoning text but never executed them.

Expired thinking settings caused API rejections on every routing request.

Conflicting refund rules made the model hesitate on clear‑cut cases, asking for customer confirmation instead.

The fix is a behavior audit for each rule:

Does it describe a real business constraint?

Is it just a patch for a past model weakness?

Does it force the model to repeat a capability it already has?

Does it conflict with other rules, tool protocols, or permission boundaries?

Keep verifiable constraints (e.g., “payment changes require second confirmation,” “production writes must pass specified tests,” “outbound emails must show recipient and summary for approval”) — these are business or safety boundaries. Drop vague exhortations like “think extremely thoroughly” or “write out every step.”

Three: The Third Waste — Treating Maximum Effort as the Default

effort

controls how much deliberation, verification, tool use, and alternative search the model invests. Low effort is faster; high effort goes deeper. But the returns curve is not linear.

On FrontierCode Diamond (50 hardest problems), Fable 5 went from 11.5% solve rate at $5.35/problem (low) to 30.9% at $19.00/problem (max) — a 2.7× capability gain for 3.5× cost. On Humanity’s Last Exam (no tools), Fable 5.1 moved from ~53% at $0.30 (low) to ~61% at $2.23 (max) — the last effort step added only ~0.5 percentage points while increasing cost 46%, within noise.

Effort mis‑calibration goes two ways:

Always high — model over‑thinks simple tasks, raising cost/latency and sometimes scattering the answer.

Always low — model stops before gathering enough evidence, skipping tool calls or checks; output looks complete but lacks foundation.

The right question: “For this task class, does failure stem from understanding, retrieval, planning, execution, or verification?”

Two calibration methods:

1. Compare stronger model at low effort vs. weaker model at high effort

On CursorBench 3.2, Fable 5.1 low effort matched Fable 5 high effort at ~1/3 the cost. Reasons: fewer steps per problem and cheaper cache reads; even at the old model’s pricing, cost was ~40% lower. The principle: model, context quality, tool availability, and effort are a combined configuration, not four independent knobs.

2. Run an effort sweep on your own task set

Take real tasks, run low/medium/high/max, record at least four metrics: success rate, average steps, latency, cost. If the performance‑cost curve flattens while evaluation hasn’t saturated, the bottleneck isn’t “think longer” — it’s tool design, data quality, permissions, retrieval structure, or task decomposition. Many teams reflexively upgrade models or raise effort when the agent simply lacks the right file, has ambiguous tool parameters, unranked search results, or missing evidence hand‑off between sub‑tasks.

Four: Real Optimization Is a Search Loop with a Holdout Set

The tutorial frames cost optimization as a small experiment system. /claude-api hillclimb splits the eval set into train and test, proposes config changes (model, effort, prompt), diagnoses failures on train, then validates on unseen holdout data.

Full customer‑support case study:

Start: Opus 4.8, default high effort.

Try Opus 5 low effort + clean forced tool rituals, scratchpad, contradictory rules → train accuracy 98.9% , cost $0.026 per ticket.

Switch to Sonnet 5 low effort → cost $0.01 , but train accuracy falls to 88.9% .

Read failed train tickets, add routing rules and refund‑limit cross‑references → train accuracy back to 98.9% .

On 14 unseen holdout tickets, final config hits 90.5% vs. original 78.6% , at ~1/5 the cost.

Without a holdout set, you risk overfitting to the cases you’ve already seen — especially when prompts grow long and rules multiply. The minimal loop for any model:

Build real task set
→ Record success rate / steps / latency / cost
→ Find biggest waste or failure pattern
→ Change one interpretable factor at a time
→ Diagnose on train samples
→ Confirm on holdout samples
→ Monitor for regression after deploy

This beats “stuff every rule into the system prompt” and “judge by a couple of demos.”

Five: Automate Cost Audits — Four Benchmarks Show Where Money Goes

cost-optimize

traces token flow from org usage reports, API usage objects, or request‑building code, then looks for savings in order: caching, input trimming, output constraints, batching. With an eval set, it also compares model/effort performance‑cost combos.

Results on four public benchmarks:

LegalBench : Shared prefix cache, low effort, Batch API → Thinking tokens 102,779 → 8,284 ; pass rate within noise; cost ~ 58% lower .

tau2‑bench retail : Explicit cache breakpoint → Pass rate flat; cost 73% lower .

OfficeQA Pro : Batching + document caching → $136.20 → $64.87 (~ 52% lower ).

SWE‑bench Verified : Medium effort, converge output to few concise sentences → Median steps 29 → 17 ; prompt tokens 75.2M → 33.7M ; cost ~ 55% lower .

These aren’t a promise that every agent can save 73% losslessly. The deeper lesson: “Big‑ticket waste sources differ by task.” Some tasks recompute the same prefix; others can’t reuse long documents; some put offline work on the real‑time path; some force the model to write intermediate explanations nobody sees; some run effort higher than needed; some have healthy caching but waste steps. Hence: measure first, optimize second. Without usage data, traces, step counts, and task evals, cost cutting becomes guesswork that slices capability.

Six: Port to Other Models & Agents — Keep Principles, Swap Implementations

The tutorial centers on Claude Platform, but its engineering logic is model‑agnostic. Translation table:

Prompt cache → Fix order of system rules / tool declarations / knowledge packs; append dynamic input; observe cache reuse via provider or custom traces.

prompt-audit → Periodically review system prompt, agent instructions, skills, tool descriptions: delete expired rituals, conflicting rules, unverifiable “try harder” demands.

Effort sweep → Include reasoning level, max turns, tool budget, retrieval depth, output length in task‑graded experiments.

hillclimb → Diagnose on train tasks, validate on holdout tasks; treat model, prompt, toolset, task routing as searchable config.

cost-optimize → Attribute cost from tokens, context length, tool calls, sub‑tasks, retries, waits, output length.

Batch API → Move evals, backfills, nightly reports, indexing, bulk extraction to async queues.

One extra principle for multi‑model agents: “Cost control and safety control must be separated.” Removing redundant “verify twice” does not cancel real payment confirmations, production write approvals, permission boundaries, or test gates. The former is repeated reasoning without new evidence; the latter are systemic safeguards against real external side‑effects. Conflating them creates the most dangerous “optimizations.”

Similarly, loading fewer tools, using smaller models for sub‑tasks, shortening output — all need boundaries. The Agent SDK notes: context accumulates in a session; stable parts (system prompt, tool schemas) can be cached; large outputs and tool definitions consume context; sub‑tasks can use isolated contexts for high‑volume exploration. Skills suit on‑demand workflows, sub‑agents for context isolation and specialization, hooks for deterministic automation. Migration principle: mechanisms differ, but stable prefixes, tool minimization, task grading, holdout‑based eval regression, and separating cost from safety governance are universal agent engineering principles (author’s independent view, not verbatim from the tutorial).

Seven: Where to Start — Three Actions Are Enough

If you maintain a working but increasingly expensive agent, don’t rewrite the framework first. Follow the tutorial’s three entry points:

Prompt audit first: Inventory prompts, skills, tool descriptions, project rules. Pull out every “must be very careful,” “check again,” “think in fixed N steps” and ask: does it measurably improve a quantifiable failure?

Cache & context audit next: Identify the largest, most repeated static prefix; lock its order; move timestamps, session IDs, transient state, and new tool results to the end; monitor hits and miss reasons.

Calibrate effort with evals last: Don’t default to max, don’t blanket‑apply low. Let real tasks tell you where diminishing returns kick in, and feed that back into routing logic.

Model capabilities change, but agent waste patterns are stable: duplicate context, bloated rules, mismatched reasoning, missing evals. Mature agent optimization doesn’t treat “saving money” as lowering intelligence; it means “every token, every tool call, every sub‑task has a reason that the results can prove.”

Sources

[1] https://claude.com/blog/reducing-cost-and-improving-performance-with-claude-platform — Anthropic: Reducing cost and improving performance with Claude Platform [2] https://docs.anthropic.com/en/docs/claude-code/skills — Claude Code skills documentation [3] https://claude.com/blog/steering-claude-code-skills-hooks-rules-subagents-and-more — Anthropic: steering Claude Code [4] https://code.claude.com/docs/en/agent-sdk/agent-loop — Claude Agent SDK: agent loop [5] https://code.claude.com/docs/en/agent-sdk/cost-tracking.md — Claude Agent SDK: cost tracking