Why Does Your AI Agent Forget Mid‑Run? Understanding Token Window Limits and Context Management

Understanding the Token Window

Large language models (LLMs) have no persistent memory; each call can only "see" the text placed in the context window , measured in tokens. Roughly 1 Chinese character ≈ 1.5 tokens, 1 English word ≈ 1.3 tokens. GPT‑4o provides a 128K‑token window, Claude 3.5 Sonnet 200K tokens (≈ 80 k–130 k Chinese characters).

During an Agent Loop, the message list grows quickly:

System prompt: ~500 tokens

Tool definitions (10 tools): ~2000 tokens

Each round (user + assistant + tool results): 500–2000 tokens

Reading a medium file: 1000–5000 tokens

Twenty rounds can already consume ~50 k tokens; large files or long test outputs can fill the window in ten rounds.

Three Manifestations of Forgetting

When the window overflows, the loss is gradual, not an immediate error:

Repetition : the agent rereads auth.ts at step 15 because the earlier read record was trimmed.

Forgotten preconditions : after 30 rounds the agent starts using Python 3.10 despite an initial instruction to use 3.11, because that instruction was evicted.

Self‑contradiction : the agent claims to have modified config.py at step 8, then later says it has not modified any config file.

All three share a dangerous trait: the agent does not realize it has forgotten.

Three Main Context‑Management Strategies

1️⃣ Sliding‑Window Truncation

Simply drop the oldest messages once a token threshold is exceeded. Example implementation:

def trim_messages(messages, max_tokens=100000):
    """Sliding window: discard earliest messages when over limit"""
    system_msgs = [m for m in messages if m["role"] == "system"]
    other_msgs = [m for m in messages if m["role"] != "system"]
    kept = []
    total_tokens = sum(estimate_tokens(m) for m in system_msgs)
    for msg in reversed(other_msgs):
        msg_tokens = estimate_tokens(msg)
        if total_tokens + msg_tokens > max_tokens:
            break
        kept.insert(0, msg)
        total_tokens += msg_tokens
    return system_msgs + kept

Pros: easy to implement. Cons: can break the required tool_use / tool_result pairing, causing the Loop to crash.

Safe variant that preserves tool pairs:

def trim_messages_safe(messages, max_tokens=100000):
    """Safe truncation: keep tool_use/tool_result pairs together"""
    system_msgs = [m for m in messages if m["role"] == "system"]
    other_msgs = [m for m in messages if m["role"] != "system"]
    safe_start = find_safe_truncation_point(other_msgs)
    trimmed = other_msgs[safe_start:]
    return system_msgs + trimmed

Suitable for short tasks where early history is not critical (e.g., a quick "find TODO" query). Not suitable when early decisions must be remembered.

2️⃣ Context Compression (Summarization)

Instead of discarding, summarize early conversation and replace it with a concise summary, preserving key information while freeing tokens.

def compress_context(messages, keep_recent=10):
    """Compress context: summarize early messages"""
    if len(messages) <= keep_recent + 1:
        return messages
    system_msg = messages[0]
    to_compress = messages[1:-keep_recent]
    recent_msgs = messages[-keep_recent:]
    summary_prompt = f"""
请将以下对话历史总结成简洁的摘要，保留：
1. 已完成的关键操作（读了哪些文件、改了什么）
2. 重要的决策和约束（技术栈、限制）
3. 当前任务的进展状态

对话历史：
{format_messages(to_compress)}
"""
    summary = call_llm(summary_prompt)
    summary_msg = {"role": "user", "content": f"[对话历史摘要]
{summary}"}
    return [system_msg, summary_msg] + recent_msgs

Claude Code’s /compact command implements this selective compression, keeping meta‑information (completed actions, current state) while discarding raw file contents.

Trade‑off: details may be lost, which can break later steps that depend on seemingly minor information.

3️⃣ External Memory (Memory Store)

Store important state outside the token window and retrieve it when needed.

Typical forms:

Key‑value store for structured state.

Vector database for semantic retrieval of unstructured data.

File system for simple persistence.

# Key‑value example
memory_store = {}

def save_memory(key, value):
    """Agent actively saves important info"""
    memory_store[key] = {"value": value, "timestamp": time.time()}

def recall_memory(key):
    """Retrieve stored info"""
    return memory_store.get(key, {}).get("value")

# Vector DB example (chromadb)
client = chromadb.Client()
collection = client.create_collection("agent_memory")

def store_memory(content, metadata=None):
    """Add document to vector store"""
    collection.add(documents=[content], metadatas=[metadata or {}], ids=[str(uuid.uuid4())])

def retrieve_relevant_memory(query, n_results=3):
    """Semantic search"""
    results = collection.query(query_texts=[query], n_results=n_results)
    return results["documents"][0]

Key challenge: the agent must know *when* and *what* to store or retrieve, which often requires explicit tool definitions and careful prompting.

Persistent Rules with CLAUDE.md

Rules placed in CLAUDE.md are re‑injected on every request, so they never get trimmed. Example project layout:

project_root/
├── CLAUDE.md   ← persistent rules
├── src/
└── ...

Typical content:

# Project conventions (never compressed)
## Technology constraints
- Python 3.11 (no 3.10 syntax)
- Use pytest, not unittest
## Prohibited actions
- Do not modify config/production.yaml
- Do not delete test files
## Compression directives
When summarizing, retain:
1. List of modified files
2. Current task progress

Guideline: cross‑task rules go into CLAUDE.md; per‑task instructions stay in the system prompt.

Session Resume

When a user closes the browser, the Agent can continue from the last point using a session_id:

def start_or_resume_session(task, session_id=None):
    """Create a new session or resume an existing one"""
    result = client.beta.messages.create(
        model="claude-opus-4-5",
        max_tokens=8096,
        system="You are a code assistant.",
        messages=[{"role": "user", "content": task}],
        metadata={"session_id": session_id} if session_id else None,
    )
    new_session_id = result.metadata.get("session_id")
    save_session_id(new_session_id)
    return result, new_session_id

The SDK stores the compressed context server‑side; on resume the same session_id re‑injects that compressed context. Because only the compressed version is restored, session resume must be paired with context compression.

Task state (e.g., which files have been migrated) also needs persistence. Example:

class TaskState:
    def __init__(self, task_id):
        self.task_id = task_id
        self.state_file = f".agent_state/{task_id}.json"
        self.state = self._load()
    def _load(self):
        if os.path.exists(self.state_file):
            with open(self.state_file) as f:
                return json.load(f)
        return {"completed_steps": [], "modified_files": [], "pending_steps": [], "session_id": None}
    def save(self):
        os.makedirs('.agent_state', exist_ok=True)
        with open(self.state_file, 'w') as f:
            json.dump(self.state, f, ensure_ascii=False, indent=2)
    def mark_step_done(self, step):
        self.state["completed_steps"].append(step)
        self.save()
    def add_modified_file(self, filepath):
        if filepath not in self.state["modified_files"]:
            self.state["modified_files"].append(filepath)
            self.save()

Register mark_step_done and add_modified_file as tools so the Agent updates the state automatically.

Real‑World Pitfall Example

Task: migrate an Express.js project to Fastify (≈ 40 route files, > 50 rounds).

Without any context management, the agent repeats file edits after round 18 and claims no migration has started after round 22.

Adding only compression reduces the issue but still loses the list of already‑migrated files, causing duplicate work.

Final solution: combine compression with an external JSON‑based memory that records migrated files, inject that list back into the system prompt before each compression step.

Result: the 50‑round migration completes without repetition.

Key Takeaways

Agent forgetting stems from the physical token‑window limit; early messages are pushed out.

Three observable symptoms: repeated actions, forgotten constraints, contradictory answers.

Three engineering approaches:

Truncation : simple but must keep tool pairs; best for very short tasks.

Compression : keeps essential information; suitable for medium‑length tasks but may drop details.

External memory : stores state outside the window; essential for long or cross‑session tasks, at the cost of added complexity.

Persist rules in CLAUDE.md to avoid them being compressed.

Session resume requires a session_id and explicit task‑state persistence.

Practical advice: start with a plain Agent Loop, add context management only when the token limit becomes a problem.