Why Prompt Caching Is More Than a Cost‑Saving Trick: It Shapes Agent Architecture

TL;DR (10 key points)

Agent cost is dominated by repeatedly reading unchanged context, not by generating answers. Prompt Cache reduces token cost, lowers latency and stabilises long‑running tasks.

Cache hit rate depends on a stable static prefix, not on the length of the dynamic tail.

Stable system prompts, tool schemas and project‑wide context dramatically improve cache usefulness.

Changing tools, models or the order of the prefix invalidates the cache.

Cache is not an optional optimisation; it reshapes the Agent’s overall architecture.

Long sessions need explicit context layering, compression and isolation. Prompt Cache, compaction, session pruning, sub‑Agents and just‑in‑time retrieval together form a complete solution.

Future Agent engineering will focus on context‑governance capability rather than raw model size.

If cache hit rates are low, audit the context structure before blaming the model.

Prompt Cache is not just a money‑saving trick

Agents that run for dozens of minutes or hours repeatedly re‑process a large, immutable block of tokens (system prompt, tool definitions, project rules). This “context tax” dominates the per‑turn cost and grows linearly with the number of turns.

The hidden cost of repeated static context

In a typical OpenClaw session the system prompt can exceed 9,600 tokens, tool schemas add another ~8,000 tokens and injected workspace files contribute several thousand more. Before any user work begins, more than ten thousand tokens are already consumed. When the conversation lasts fifty turns, the repeated processing of this static prefix becomes a major expense.

What Prompt Cache actually stores

Prompt Cache does not cache raw text. It caches the model’s computed static prefix state. An Agent request can be split into two parts:

Static prefix : system instructions, tool schemas, long‑term project rules – the expensive part that can be reused.

Dynamic tail : user messages, tool outputs, observations and any newly added context for the current turn – the part that must be recomputed each round.

Layered context cost model

Think of an Agent’s context budget as three layers:

Fixed overhead – system prompt, tool schemas, skill lists. This layer is the most cache‑friendly (⭐️⭐️⭐️).

Semi‑fixed overhead – files such as CLAUDE.md, memory snapshots, project contracts. When stable, they can also be cached (⭐️⭐️).

Dynamic overhead – conversation history, file contents, tool outputs. This layer grows each turn and cannot be cached (⭐).

Maximising stability in the first two layers raises cache hit rates, while the third layer must be actively managed through compression and pruning.

Why cache influences system architecture

Cache stability forces the prefix to be immutable. Any runtime modification of system prompts, tool definitions or the order of sections breaks the cache, turning the most cache‑friendly content into a liability. OpenClaw, for example, rebuilds the system prompt each run with a deterministic order (Tooling → Safety → Skills → Workspace → Runtime) and isolates all mutable state to later layers.

Tools should be small, well‑defined and rarely changed. Large, mutable tool sets increase prefix volatility and hurt cache efficiency.

Three core engineering rules

1. Keep the persistent prefix short and stable

Avoid modifying system prompts, tool definitions or long‑term rules at runtime. Stability in the prefix enables effective caching.

2. Move state changes to the dynamic tail

Place current phase, progress, temporary reminders and observations in the message layer, memory files or external storage rather than in the system prompt. OpenClaw stores daily logs in memory/YYYY‑MM‑DD.md and long‑term decisions in MEMORY.md, retrieving them on demand via memory_search or memory_get.

3. Keep the toolset minimal and predictable

Include only necessary tools and ensure each tool’s output is concise. Stable tool schemas reduce cache churn.

From Prompt Engineering to Context Engineering

Effective context engineering treats the context as a finite resource that must be designed, constrained and governed. Anthropic’s 2025 article “Effective context engineering for AI agents” summarises the shift: keep the context minimal, high‑signal, and retrieve additional information just‑in‑time.

OpenClaw’s end‑to‑end context governance pipeline

Fixed system prompt – rebuilt each run with a deterministic order.

Skills on‑demand – only metadata is stored in the prompt; full skill definitions are read from SKILL.md when needed.

Session pruning – before a cache write after TTL expiry, old tool results are soft‑pruned (replaced with ...) or hard‑cleared, reducing the payload for the next cache.

Compaction – when the window nears its limit, OpenClaw runs a silent memory‑flush, writes important notes to disk, then summarises early conversation into a compact abstract.

Memory write‑before‑compaction – ensures critical notes are persisted before summarisation.

Multi‑Agent isolation – each sub‑Agent has its own workspace, session and authentication, preventing noisy investigations from polluting the main context.

These mechanisms directly improve cache hit rates and keep long‑running agents affordable and responsive.

Eight practical actions to improve your Agent

Shorten and stabilise the persistent prefix (system prompt, tool definitions, long‑term rules).

Move dynamic state (phase, progress, observations) to the message or memory layer.

Keep the toolset minimal and ensure each tool returns short, clear results.

Adopt a just‑in‑time retrieval strategy: store file paths or references and read the full content only when needed.

Enable automatic compaction for long sessions to summarise old history.

Delegate noisy or exploratory tasks to sub‑Agents and only return concise summaries.

Monitor cache‑specific metrics ( cache_creation_input_tokens, cache_read_input_tokens, input_tokens) rather than total token count.

If cache hit rates remain low, first audit the context structure (prefix stability, tool volatility, history size) before tweaking the model.

Monitoring and debugging cache behaviour

/context list

– shows current window utilisation and injected file sizes. /context detail – breaks down token usage per tool schema and skill entry. /usage tokens – provides per‑reply token usage details. /status – reports total session tokens and estimated cost.

When cache miss persists, check whether the prefix has drifted, tools have changed, or the history has grown unchecked before considering a model upgrade.

References

Akshay Pachaar, “Prompt caching, clearly explained”, https://x.com/akshay_pachaar/status/2031021906254766128

Anthropic, “Prompt caching”, https://platform.claude.com/docs/en/build-with-claude/prompt-caching

Anthropic, “Effective context engineering for AI agents”, https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

Anthropic, “Building agents with the Claude Agent SDK”, https://claude.com/blog/building-agents-with-the-claude-agent-sdk

Thariq, “Lessons from Building Claude Code: Prompt Caching Is Everything”, https://x.com/trq212/status/2024574133011673516

OpenClaw documentation: Context – https://openclawcn.com/docs/concepts/context/

OpenClaw documentation: Compaction – https://openclawcn.com/docs/concepts/compaction/

OpenClaw documentation: Session Pruning – https://openclawcn.com/docs/concepts/session-pruning/

OpenClaw documentation: Memory – https://openclawcn.com/docs/concepts/memory/

OpenClaw documentation: System Prompt – https://openclawcn.com/docs/concepts/system-prompt/

Illustrative diagrams