Adding Memory: Enabling Multi‑Turn Conversations in an LLM Agent

One‑Sentence Conclusion

Practical 1 upgrades the raw messages list to a ContextManager so that all user and assistant turns are added only via append_user / append_assistant , and the model receives a single payload from build_for_llm() . When the token budget is exceeded the manager slides a window to keep the most recent N user turns while preserving the system prompt and reserving hooks for future tool handling.

Goal and Acceptance Criteria

Provide append_user, append_assistant, and build_for_llm methods.

Estimate token usage with tiktoken (fallback cl100k_base) and expose a configurable CONTEXT_TOKEN_BUDGET.

When the budget is exceeded, retain only the most recent CONTEXT_KEEP_RECENT_TURNS user turns (and their following assistant messages).

Remove direct manipulation of the raw messages list from the CLI.

Reserve append_tool_* hooks for later practical steps.

Acceptance Tests

Run 20+ dialogue turns without OOM or unbounded slowdown; after truncation the retained recent turns remain answerable.

Terminal displays an approximate token count ( ctx≈) and increments a compact× counter each time truncation occurs.

On error, calling rollback_last_user() restores the history without leaving unanswered user queries. build_for_llm() returns a list whose first element is the system message, followed by alternating user/assistant dicts.

File Changes and Repository Layout

mini-agent/
├── .env.example          # + CONTEXT_TOKEN_BUDGET, CONTEXT_KEEP_RECENT_TURNS
├── config.py             # context‑related configuration
├── context.py            # ★ new file implementing ContextManager
├── brain.py              # chat() now accepts payload with system prompt
├── main.py               # switched to use ContextManager
└── requirements.txt      # + tiktoken

Git: incremental commit on top of article-00 creates tag article-01.

Core Code – context.py

@dataclass
class ContextManager:
    system_prompt: str
    token_budget: int
    keep_recent_turns: int = 10
    messages: list[dict] = field(default_factory=list)

    def append_user(self, text: str) -> None: ...
    def append_assistant(self, text: str) -> None: ...
    def build_for_llm(self) -> list[dict]:
        self._maybe_compact()
        return [{"role": "system", "content": self.system_prompt}, *self.messages]

Design Highlights

System prompt excluded from messages : the system prompt lives in DEFAULT_SYSTEM_PROMPT and ContextManager.system_prompt; build_for_llm is the sole place that concatenates the system message.

Token estimation : estimate_tokens uses tiktoken (fallback to cl100k_base) for a closer approximation to actual API usage.

Sliding window : counts back from the last user turn, keeps that turn and all following assistant messages, preventing a round from being split.

Summarizer hook : optional callable that can compress discarded history into a summary (enabled in later practical steps).

Tool hooks : append_tool_* placeholders will ensure tool calls follow the same compact logic when activated.

Usage in main.py

ctx = ContextManager(
    system_prompt=DEFAULT_SYSTEM_PROMPT,
    token_budget=settings.context_token_budget,
    model=settings.model,
    keep_recent_turns=settings.context_keep_recent_turns,
)
ctx.append_user(user_input)
result = brain.chat(ctx.build_for_llm())
ctx.append_assistant(result.text)

If an error occurs, call ctx.rollback_last_user() to mimic the behavior of the original implementation.

Common Pitfalls

Compact causes "forgetting" : early turns are deliberately dropped; increase CONTEXT_KEEP_RECENT_TURNS or CONTEXT_TOKEN_BUDGET, or enable the summarizer (Practical 5) to retain long‑term facts.

Truncation may break tool blocks : currently no tool calls; Practical 2 will ensure that assistant tool_calls + all tool results stay together, and Practical 3 will switch to a ReAct‑style step‑wise cut.

tiktoken vs real usage : the displayed ctx≈ is a local estimate; the API’s prompt_tokens may differ by 10‑20 % and should be used for billing.

System message removal : _maybe_compact only mutates self.messages, never the system entry, so the system prompt remains the first element.

Avoid direct append on messages : all modifications must go through the ContextManager to keep the "single‑exit" invariant.

Mapping to Theory (Agent Series Module 2)

Message structure : user/assistant dicts stored in ContextManager.messages.

Single export : ContextManager.build_for_llm is the only function that produces the payload for the LLM.

Token rough estimate : implemented with tiktoken in estimate_tokens.

Sliding window : realized by keep_recent_turns and the private method _compact_sliding_window.

Tool tracking placeholder : interface reserved via append_tool_result.

Summarizer hook : reserved callable summarizer for future LLM‑based compression.

Framework Choice (Series‑Level)

The current steps (0‑2) use a pure OpenAI SDK plus custom modules, matching the original Agent series outline and suitable for teaching. Alternatives such as LangGraph (step 3), Pydantic AI (step 2), or a hybrid approach are listed for later migration.

Next Article Preview

Practical 2 will introduce the first tool call (calculator & time), adding a ToolRegistry with JSON Schema, extending Brain.chat with a tools argument, and enabling ContextManager.append_tool_result.

Summary

Delivered ContextManager with token budgeting and sliding truncation; CLI no longer manipulates the raw messages list. build_for_llm() is now the single entry point before invoking the LLM, and future tool, compact, and trace features will attach here.

Acceptance: multi‑turn dialogue works, token count visible, over‑budget truncation triggers, and errors roll back correctly.

Tag article-01 completes Practical 1; subsequent steps will build on this foundation.