Why Agent Conversations Aren’t Just Chat Logs: Effective Context Management

Conversation Is Not a Chat Log

Agent context should be assembled as a "current world state snapshot" based on role contracts, tool trajectories, and window budgeting. If the snapshot is mis‑managed, even a powerful model will forget instructions, repeat tool calls, or lose track after many rounds.

Problem Scenario: Why Multi‑Turn Demos Crash

System rules get forgotten – the context becomes too long and truncation drops early system messages.

Tool‑call alignment breaks – assistant emits tool_calls without matching tool messages, or tool_call_id mismatches cause hallucinated returns.

History grows linearly – each round sends the full conversation to the API, exploding token cost and latency.

Irrelevant data leaks – raw HTTP responses, debug logs, or error stacks waste tokens and expose implementation details.

The root cause is usually the lack of a dedicated context‑management layer that decides who appends, who trims, what counts as a round, and how tool trajectories are atomically written back.

Position in the Agent Architecture

The short‑term memory layer sits between the user input/history and the large model. After the model produces a tool_calls response, the tool execution result is written back by the observer , forming a closed loop. The orchestration layer (module 5) determines when to invoke the brain, while the context layer decides what the brain sees.

Core Concept 1: Message Structure

system : identity, boundaries, tool principles – keep short and stable; large knowledge bases belong elsewhere (RAG or module 8).

user : raw user input and optional structured supplements – never place secret policies here (risk of injection, module 10).

assistant : natural‑language reply, optionally with tool_calls – when tool_calls are present, content is usually empty; do not manually corrupt the JSON.

tool : execution result – must include the matching tool_call_id so each call pairs with its result.

Core Concept 2: What Belongs in history

The history should serve the model’s next decision, not act as a full audit log.

Include: current user intent (merged duplicates), assistant’s final answer visible to the user, complete tool trajectory (calls + results), compressed tool results that retain decision‑relevant fields, and the current plan/TODO (as a scratchpad or incremental system message).

Exclude: raw API JSON/HTML payloads, repeated failed tool calls (keep only the last attempt and error count), full chain‑of‑thought when policy disallows, other users’ sessions (privacy), massive RAG chunks (use as temporary context with reference IDs).

Core Concept 3: Context‑Window Strategies

Sliding window : keep system + the most recent N rounds of user/assistant/tool. Simple but may drop early constraints or completed sub‑tasks.

Summarization compression : run an LLM to compress older turns into a single summary message. Preserves long‑range semantics at the cost of an extra LLM call and possible detail loss.

Hierarchical memory : hot data stays in history, cold data moves to a vector store (module 7). Scalable but adds engineering complexity and can cause “false amnesia” if retrieval fails.

Recommended retention order (hardest to drop → easiest):

1. system (core contract, unless version upgrade)
2. recent K full tool trajectories (K≈2‑5 depending on task depth)
3. current user/assistant messages relevant to the task
4. older dialogue summarized into a single message
5. discard: completed sub‑task attempts, duplicate tool error details

Core Concept 4: Token Estimation and Cost

Agent runs cost more than a plain chat because each step adds history, tool results, and possibly a new assistant message. Rough cost formula:

task_cost ≈ Σ (input_tokens + output_tokens) × price_per_token
input_tokens ≈ system + current history + tool definitions (if sent each step)

Optimization ideas (ordered by impact):

Slim tool results – send only essential fields (e.g., status, preview, byte size) instead of full payloads.

Avoid resending unchanged tool schemas – cache them if the API supports session‑level registration.

Incremental append only – prevent deep‑copy duplication of old messages.

Early stopping and step limits – enforce max_steps to cap token usage.

Use a cheap model for summarization – compress history with a smaller LLM while the main loop runs the primary model.

Observation‑driven profiling – record input_tokens each step, identify the bulkiest tool, and optimise its payload.

Token estimation tip: Chinese ≈ 1‑2 characters per token, English ≈ 4 characters per token; always calibrate with the real tokenizer or API usage fields.

Core Concept 5: Minimal ContextManager Contract (Pseudocode)

class ContextManager:
    def __init__(self, system_prompt, token_budget, summarizer=None):
        self.system = system_prompt
        self.messages = []  # user / assistant / tool, no system
        self.token_budget = token_budget
        self.summarizer = summarizer

    def append_user(self, text):
        self.messages.append({"role": "user", "content": text})

    def append_assistant(self, response):
        # response from Brain.decide: text and/or tool_calls
        self.messages.append(build_assistant_message(response))

    def append_tool_results(self, calls, results):
        # atomic write‑back: each call.id has a matching result
        for call, result in zip(calls, results):
            self.messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": serialize(result),
            })

    def build_for_llm(self):
        if estimate_tokens(self.system, self.messages) > self.token_budget:
            self._compact()  # sliding / summarization, keep tool blocks atomic
        return [{"role": "system", "content": self.system}] + self.messages

    def _compact(self):
        # keep system; summarize oldest 50% of rounds; keep recent full tool blocks
        old, recent = split_messages(self.messages, keep_recent_tool_rounds=3)
        summary = self.summarizer.summarize(old) if self.summarizer else None
        self.messages = ([summary_message(summary)] if summary else []) + recent

Orchestration usage example:

ctx.append_user(user_input)
for step in range(max_steps):
    messages = ctx.build_for_llm()
    result = brain.decide(messages, tools=tools)
    ctx.append_assistant(result)
    if result.kind == "tool_calls":
        results = [tools.execute(c) for c in result.calls]
        ctx.append_tool_results(result.calls, results)
        continue
    if result.kind == "final_answer":
        return result.text

Key takeaways: keep append and compact separate; build_for_llm is the single entry point to the model, preventing scattered message deletions.

Design Checklist Before Launch

Is the system prompt short and stable, and separated from history‑bloat sources?

Do tool_calls and tool results match one‑to‑one? Are all parallel calls written back?

Are string‑concatenated pseudo‑rounds prohibited? Is the native role structure used?

Is the over‑window strategy explicit (sliding, summarization, hierarchical) rather than relying on SDK defaults?

Does truncation preserve whole tool blocks? Does summarization retain unfinished tasks and key facts?

Are tool results slimmed; large payloads stored only in trace logs?

Is a token upper bound estimated per task and linked to max_steps and retry limits?

Are sessions isolated – no mixing of messages from different users?

Are compact events logged for debugging?

Is intermediate reasoning kept out of user‑visible history to satisfy compliance?

Relation to Practical Modules

Message structure and multi‑round appends – Module 1 (Brain) and Module 5 (Orchestration).

Tool trajectory write‑back – Module 3 (Tool layer).

Repeated build_for_llm in the main loop – Module 5 (ReAct loop).

Summarization and long‑term memory – Module 7.

Token tracking and tracing – Module 11.

Conclusion

Context is a structured set of messages that serves the next decision, not a raw chat export.

Tool trajectories must be atomic and paired; this is the minimal requirement for a stable Agent loop.

When the context window is exceeded, proactively slide, summarize, or tier memory, watching out for “Lost in the Middle” of tool blocks.

Token cost grows with steps and tool payloads; slimming tool results often beats upgrading the model.

Using a dedicated ContextManager to unify append, compact, and build_for_llm keeps the brain and orchestration layers stable.