Beyond Caching: Architectural Solutions for Agent Context Compression Losing Tool Calls

Interview Scenario: The Cache Patch Isn't Enough

During a second-round interview for an Agent system design role, the interviewer posed a scenario: an Agent running a long task exceeds the context window, so compression is applied. However, compression drops part of the tool call history, leading the model to repeat the same tool calls. The candidate's first answer was to add a cache storing tool results and returning them on duplicate calls. The interviewer shook his head, calling it a patch, not a design, and asked why the loss happens, what exactly is lost, and how to prevent it at the root.

Why Compression Loses Tool Call Details

Compression is inherently lossy. The article identifies three distinct leakage points:

Summary Generation Phase : Most compression schemes call a model to summarize history. Unless the prompt explicitly demands preserving tool call details, the model defaults to high-level descriptions (e.g., "queried user order") instead of concrete facts (e.g., query_order(order_id=8823) returned shipped).

Sliding Window Truncation : A cruder approach simply discards the earliest N conversation turns. This removes entire segments wholesale; the model has zero awareness of the dropped content.

Attention Dilution in Long History : Even if information remains in the context, without structured markers it gets buried in a long dialogue. The model may fail to attend to it during decision-making.

The cache-only answer only mitigates the downstream effect of the third issue, ignoring the first two root causes.

Core Design Principle: Separate State from Narrative

The key insight: conversation history is a narrative for humans; Agent execution state is structured data for machines. They should not share the same compression logic.

1. Maintain an Independent "Action Ledger"

Do not rely on compressed summaries to remember what was done. Keep a separate, structured ledger outside the conversation context that is never compressed. Each entry records:

Tool name

Key parameters (or parameter hash)

Execution result highlights

Timestamp

Execution status (success/failure/in-progress)

This ledger is injected as an independent segment before every decision round. Even if the dialogue history is compressed to a few sentences, the ledger remains complete — effectively giving the Agent a separate "operation log" distinct from the "chat log."

2. Idempotent Interception at the Execution Layer

This is the safety net for when the model still issues duplicate calls due to memory loss. Generate a signature for each call: function name + hash of parameters. Maintain a set of executed signatures:

Signature hit → return cached result, skip re-execution.

Signature miss → allow execution, then record signature and result.

For side-effect operations (sending email, placing orders, charging, writing to DB), this is a hard requirement. For read-only operations (queries, search), it mainly saves resources and cost.

3. Explicitly Require Tool Call Preservation in Compression Prompts

If summarization compression is used, the prompt must explicitly instruct:

"Please fully preserve every tool call's name, key parameters, and result highlights; do not generalize."

Models follow instructions closely; a clear retention checklist dramatically reduces the chance of critical details being erased during compression.

4. Externalize Storage, Retrieve on Demand

A more radical approach: don't make the model "remember" tool history; make it "queryable." Store full records externally (database, files, vector store). When the model needs to know "have I done this?", it actively retrieves. This decouples memory capacity from the context window size — memory can grow indefinitely while the context carries only the currently relevant slice.

5. Hierarchical Rolling Summaries Instead of One-Shot Compression

For long dialogues, use rolling summaries: periodically condense a segment's key actions into a few lines, then progressively aggregate multiple segment summaries into higher-level summaries. Compared to accumulating a large history and compressing once, this gradual compression loses less information per step, increasing the probability that key facts survive through the layers.

Synthesized Answer

"Duplicate calls stem from compression dropping tool call details. The fix isn't just an execution-layer patch; it requires architecturally separating the Agent's execution state from the conversational narrative. Concretely: maintain an independent, non-compressed action ledger recording every call's essential info; add an idempotent interception layer keyed on function+parameter signatures to guard side-effect operations; if compression is unavoidable, mandate explicit retention of call details in the summarization prompt; long-term, externalize call history for on-demand retrieval instead of passively relying on the context window."