Why Agent Context Management Prioritizes Information Over Shortening Prompts

1. What the Agent Context Contains

The Agent’s context is not a single prompt string but a multi‑layer information stack. Each layer has distinct sources, update frequencies, and management strategies.

Layer 1 – System Instructions (static, defined by the developer, unchanged per session).

Layer 2 – Tool Definitions (semi‑static JSON schemas of available tools; a full‑featured developer Agent may register dozens of tools, consuming thousands of tokens).

Layer 3 – Knowledge Injection (dynamic external retrieval such as project docs, code snippets, RAG results, business rules; size and source vary per request).

Layer 4 – Memory (persistent facts and session‑level summaries; stored after summarisation or structuring).

Layer 5 – Dialogue Flow (user messages, model replies, tool calls and results; grows with each turn).

Effective context management must address all five layers while balancing capacity, content, structure, and lifecycle.

2. Four Management Dimensions

The context can be viewed as a budget‑constrained information space. Four dimensions answer four fundamental questions.

Capacity Management – How many tokens can be stored? For a 200 K token window, ~184 K tokens remain for input after reserving 16 K tokens for output; approaching the limit destabilises inference.

Content Management – What should be stored? Not every retrieved piece is useful; e.g., out of 20 RAG fragments only a few may be relevant, and stale tool‑call results can interfere with the current task.

Structure Management – How should items be ordered? Placement influences the model’s attention; the “Lost in the Middle” effect means middle‑positioned information is often ignored.

Lifecycle Management – When to replace or discard? Decisions include how often to summarise dialogue, whether to load tools on demand, and whether memory updates are synchronous or asynchronous.

These dimensions are interdependent; a change in one often impacts the others.

3. Typical “Context Out‑of‑Control” Scenarios

Scenario 1 – Long‑dialogue Pollution

A user discusses project A for 20 turns, then switches to unrelated project B. The earlier 20 turns remain in context, causing the model to mix project A details into answers for project B. The root problem is improper content management – outdated context was not removed.

Scenario 2 – Tool‑Definition Bloat

An “all‑round” developer Agent registers 80 tools. Each tool’s JSON schema averages 200 tokens, consuming 16 K tokens per request (≈8 % of a 200 K window, 12.5 % of a 128 K window). This static load reduces space for dynamic content, illustrating failures in capacity and lifecycle management.

Scenario 3 – RAG Retrieval Dilution

A query about an API’s rate limit retrieves ten document fragments, three of which describe different API versions, including a deprecated one. The model must internally select the correct version, exposing a mismatch between retrieval relevance and reasoning usefulness. The issue stems from content and structure mis‑management.

4. Four Practical Baselines

Baseline 1 – Explicit Budgeting

Allocate token quotas per layer (e.g., system instructions ≤5 K, tool definitions ≤15 K, knowledge injection ≤10 K, dialogue history ≤50 K) and check the budget before each request. Adjust proportions per task but maintain awareness of the overall token budget.

Baseline 2 – Static / Semi‑Static Separation + Cache Reuse

Separate static parts (system instructions, tool definitions, project rules) from dynamic parts (user messages, tool results, RAG fragments). Place static content in a prompt prefix and leverage provider‑side prompt‑caching (e.g., Anthropic up to 90 % savings, OpenAI up to 50 %).

Baseline 3 – Selective Forgetting Over Brutal Truncation

When the dialogue exceeds the budget, prefer asynchronous summarisation of early turns or explicit context‑slice markers instead of naïvely keeping only the most recent N turns.

Asynchronous summarisation: Run a small model in the background to generate a structured summary of early dialogue (what was done, current state, key decisions) and replace the raw messages.

Context‑slice markers: Insert explicit “topic‑switch” markers so the model knows a previous segment is no longer active.

Result deduplication: For large tool results, keep a concise summary. Example: read_file returns 500 lines (~3 000 tokens); later keep only “read_file returned src/main.py with function signatures” (~200 tokens), saving ~2 800 tokens.

Baseline 4 – Inter‑Layer Isolation

Prevent higher‑level layers from being polluted by lower‑level data.

XML‑like tags: Wrap each layer with non‑semantic tags such as <system>, <tools>, <context> to make boundaries explicit.

Tool result validation: Before inserting a tool’s output into the context, check for error signals and wrap the result in a uniform format (success/failure/empty) to avoid raw error messages contaminating the model’s view.

Knowledge‑injection denoising: Apply rerank, deduplication, and version tagging to RAG fragments before injection, ensuring the model sees the current correct version rather than an unordered stack of historical versions.

5. Maturity Ladder for Context Management

L0 – Free Heap: All information concatenated without order or token limits; everything is unmanaged.

L1 – Truncate‑First: Keep only the most recent N turns; manages capacity but ignores content, structure, and lifecycle.

L2 – Summarisation: Replace raw history with summaries; adds content awareness but still lacks structure and lifecycle handling.

L3 – Budget + Layering: Explicit token budgeting, static/dynamic separation, and clear boundaries; covers capacity, content, and structure.

L4 – Full Lifecycle: On‑demand tool loading, asynchronous memory writes, versioned context, automatic compression; addresses all four dimensions, though threshold tuning remains an engineering challenge.

Most current Agent systems sit at L1 or L2; reaching L3 often requires a mature Agent framework, while L4 is an emerging engineering goal.