Why a 92% Prompt Cache Hit Rate Slashes LLM Costs: A Deep Dive into Context Engineering

Last month I wrote about Prompt Cache and noted that the real difficulty in managing agents lies in context, not isolated prompt tricks. Recent deep‑dive posts by Avi Chawla, Anthropic’s official docs, and Thariq’s notes reveal that Prompt Caching is now an architectural discipline rather than a simple optimization.

TL;DR

Claude Code achieves a 92% cache‑hit rate by cleanly separating stable prefixes from dynamic tails.

KV Cache reuses the model’s intermediate K/V tensors, not previous textual answers.

Automatic caching is now a first‑class feature; TTL defaults to 5 min (optional 1 h) and look‑back covers ~20 blocks.

Cache hit rate is calculated as

cache_read_input_tokens / (cache_read_input_tokens + cache_creation_input_tokens)

.

Effective Prompt Caching is the foundation of Context Engineering.

Understanding the 92% Figure

Claude Code’s team posted a tweet titled Prompt Caching Is Everything. Running long‑task agents quickly shows two failure modes: (1) recomputing the entire context each round, and (2) context length growing unbounded, inflating both cost and quality loss. When a session runs dozens or hundreds of rounds, caching becomes a prerequisite, not an after‑thought.

Thariq’s post breaks Claude Code’s cache layout into three layers:

Global layer (stable prefix) : system prompts and tool definitions, shared across projects and sessions.

Project layer (stable prefix) : the CLAUDE.md file and project‑specific conventions, reused within the same project.

Dynamic tail : task‑specific history and tool outputs that grow with the conversation.

Only the stable layers are cached; the dynamic tail is appended and re‑hashed each round.

What Prompt Caching Actually Reuses

Many assume Prompt Caching stores the previous answer for reuse. In reality it caches the intermediate K/V tensors computed during the pre‑fill phase. Since each token’s Key and Value depend only on preceding tokens, an identical prefix can be fetched directly without recomputation.

The cache lookup uses a cryptographic hash of the exact token sequence, not semantic similarity. Anthropic’s docs require the prefix order tools → system → messages. Any change—e.g., swapping "1 + 2 = 3" with "2 + 1 = 3"—breaks the hash and forces a miss.

Cost Implications

LLM inference consists of two cost‑heavy stages:

Prefill : dense matrix multiplication for the full input, the most expensive operation.

Decode : token generation using the already‑computed state, relatively cheap.

Prompt Caching eliminates repeated prefill work. However, cache writes cost 1.25× the base rate, while reads cost only 0.1×. High hit rates are therefore essential for the economics to work.

Avi’s calculation assumes a 20 k token stable prefix processed over 50 rounds, totaling 1 M tokens of redundant prefill. With caching, the same workload drops from roughly $6 to $1.15, an 81% cost reduction.

Cache Warm‑up Over a 30‑Minute Session

Breaking the session into minutes illustrates the dynamics:

Minute 0 : Load system prompts, tool definitions, and CLAUDE.md (≈20 k tokens). This is the most expensive moment, paid once.

Minutes 1‑5 : Agent explores files; dynamic tail grows, but the stable prefix is read from cache at $0.30/MTok instead of $3.00/MTok.

Minutes 6‑15 : The Plan sub‑agent receives a concise summary, keeping the dynamic tail small; cache hit rate climbs above 90% and TTL refreshes.

Minutes 16‑25 : New requirements add more tool output, yet the 20 k token base remains cached.

Minute 28 : A /cost check shows a raw cost of 2 M tokens at full rate versus 1.15 M after caching—an 81% drop.

Two key observations:

Summarizing tool output (instead of feeding raw logs) prevents the dynamic tail from ballooning.

Cache stays hot only if requests arrive within the TTL; a pause >5 min forces a cache miss.

Why the 92% Hit Rate Holds

Anthropic’s documentation stresses three disciplined practices:

Maintain order : The prefix must follow tools → system → messages. Any reordering changes the hash and invalidates the cache.

Keep prefixes clean : Avoid inserting timestamps, random IDs, or mutable tool definitions into the system prompt.

Append state changes : Updates belong at the end of the message list, never in the stable prefix.

Violating any of these rules quickly degrades hit rates, as shown by the “cache‑break” diagram (image omitted).

Often‑Overlooked Details

Automatic caching is now default; you only need a top‑level cache_control and the system places the breakpoint at the last cacheable block.

Look‑back limit : Automatic look‑back covers ~20 content blocks. If the conversation grows faster, you may need explicit breakpoints.

Minimum cacheable length varies by model (e.g., Claude Opus 4.5/4.6/4.7 → 4096 tokens, Sonnet 4.6 → 2048 tokens, etc.). Short prefixes below the threshold won’t be cached.

Concurrency pitfall : Cache entries become readable only after the first response starts. Parallel sub‑tasks must wait for that point.

From Prompt Caching to Full‑Blown Context Engineering

Prompt Caching solves the "don’t recompute stable content" problem. Context Engineering expands this by deciding which content stays stable, which is fetched on demand, which is compressed, and which is omitted entirely.

Anthropic’s "Effective context engineering for AI agents" outlines that context is a limited resource; the goal is to keep only the highest‑signal tokens.

Seven Practical Checks for Existing Agents

Visualize the prefix hierarchy (global, project, session) to spot cost hotspots.

Strip dynamic noise (timestamps, random IDs) from the stable prefix.

Offload heavy baseline material ( CLAUDE.md) into Skills for on‑demand loading.

Keep the tool set minimal and non‑overlapping; defer tool definitions until actually called.

Route verbose outputs to hooks or sub‑agents, feeding only summaries to the main session.

Monitor cache‑related usage fields: cache_creation_input_tokens, cache_read_input_tokens, and input_tokens. Set alerts on hit‑rate drops.

Treat compaction as an architectural action: append compression commands as new messages without altering the stable prefix.

Applying these steps typically yields noticeable cost reductions because the system stops paying for redundant prefill work.

Conclusion

Prompt Caching has evolved from a niche prompt‑trick into a core engineering discipline that reveals deeper context‑management principles. By enforcing stable‑prefix ordering, cleaning dynamic noise, and monitoring cache metrics, you turn a 92% hit rate into predictable cost and latency savings, enabling reliable long‑task agents.