Skip to content
Context Engineering: Why More Context Makes Agents Worse — And 5 Fixes

Context Engineering: Why More Context Makes Agents Worse — And 5 Fixes

This article explains why excessive context degrades AI agent performance, citing research on 'lost in the middle' and context rot, and presents five context engineering techniques — prune, order, retrieve, compress, delegate — with a practical CONTEXT.md template and a customer support case study reducing tokens from 180K to 7K and cost from $0.54 to $0.02 per reply.

ThinkInAI Community
ThinkInAI Community
ThinkInAI Community

Introduction: The Context Paradox

Google let Gemini play Pokémon for hundreds of hours. The longer it played, the worse it performed. After roughly 100,000 tokens of history, the agent stopped making new plans and simply repeated past actions. The model and game didn't change — only the amount of context the agent had to read before each action grew. Most users do the opposite: they paste entire documents, full chat histories, every tool, and every rule, then expect the agent to find the one sentence that matters. Usually it can't, and the more you give, the less it actually uses.

Why More Context Hurts

Information in the Middle Gets Lost

In the "Lost in the Middle" study, Stanford researchers placed the only document containing the answer at different positions in a long prompt. Models performed best when the document was at the start or end; when buried in the middle, GPT-3.5 performed worse than having no document at all.

Longer Inputs Degrade Reliability Even for Fixed-Difficulty Tasks

Chroma tested 18 models — including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3 — on tasks of identical difficulty. The only variable was input length. As input grew, answer reliability dropped.

Agents Start Copying Their Own Past Behavior

Google's Gemini 2.5 report describes the Pokémon agent: beyond 100k tokens it relied on long-term history, repeating old actions instead of planning anew. Anthropic calls this "context rot" and treats the context window as an attention budget: every added token consumes a slice.

The core insight: the context window is a budget, not a container. A larger window just lets you pay more for the same mistake.

Five Key Context Engineering Actions

1. Prune: Keep Only the Minimum High-Signal Tokens

Anthropic's principle: use the fewest high-signal tokens to get the result you want. Three categories almost never get pruned:

System prompts — cover every possible situation, most of which never occur in the current task.

Tools — the task needs 6, but 40 are provided. Anthropic's rule: if a human engineer can't say which tool to use, the agent can't either.

Examples — ten edge-case examples teach the model edge cases, not the task.

Principle: every line in context must prove its usefulness for the current step, not for some future step.

2. Order: Put Critical Content at the Start and End, Never in the Middle

Position determines attention. Put immutable rules at the top — they're read first and stay cached. Put the current goal and next step at the bottom, right before the agent acts. Manus (acquired by Meta) uses a continuously rewritten todo.md so the plan always sits at the context end, never sinking into the middle during a 50-step task.

Principle: anything the agent must never forget goes either at the start or repeated at the end.

3. Retrieve: Give the Agent a Map, Not the Whole Library

Old approach: preload everything — entire codebase, whole help center — just in case. Better: just-in-time retrieval. The agent keeps only lightweight references — file paths, links, search queries — and loads actual content only when a step truly needs it. Anthropic describes how Claude Code handles large codebases: it searches and uses head / tail to view file slices instead of stuffing whole files into context. Live retrieval also eliminates stale information: a document pasted in March stays March; a search today returns today's reality.

Principle: paste the map; let the agent fetch what the map points to.

4. Compress: Keep Notes, Not Full Transcripts

Long tasks inevitably fill the window, so content must be removed.

Discard raw tool outputs — once used, the agent doesn't need to re-read 4,000 lines; keep only the conclusion drawn.

Keep a notes file — Anthropic's own Pokémon-playing Claude kept running notes: steps trained on a route, levels still needed. Even after context reset, it could resume the plan. Same method works for any long task: let the agent read/write a NOTES.md outside the context window.

Keep failures — Manus found the opposite of common practice: leave failed actions and their errors in context. The model sees what went wrong and won't retry; if you erase them, it hits the same wall again.

Principle: compress successes into notes; keep failures verbatim in view.

5. Delegate: One Window, One Job

When a task requires reading large volumes, hand the reading to a sub-agent with a clean context window. In Anthropic's research system, each sub-agent can spend tens of thousands of tokens exploring, then return a compressed summary of 1,000–2,000 tokens. The main agent never sees the messy exploration, only the result. This is where delegation pays off: delegate reading, not decision-making. Decisions stay in one window.

Principle: if you need to read 50 pages for a one-paragraph answer, let a separate window read the 50 pages.

Worked Example: A Customer Support Inbox

Scenario: an agent replies to customer emails, with a 240-article help center, three years of tickets, and a policy change in March shortening refunds from 30 to 14 days.

Naïve approach : paste everything every call — entire help center, full customer history, 60 lines of rules — ~180k tokens. Result: after 200 emails in a week, 31 still promised 30-day refunds because old articles were still in context and longer.

Context-engineered approach :

Prune : keep 12 rules that apply to every email; move the rest to searchable articles.

Order : rules at top, customer email and reply goal at bottom, current refund policy pinned right above the email.

Retrieve : search help center, fetch max 3 articles per email, newest version only.

Compress : condense customer history into 10-line notes instead of loading 40 raw tickets.

Delegate : a sub-agent reads verbose order logs and returns a 5-line summary.

Result: per-reply context dropped from ~180k to ~7k tokens; policy-reference errors fell to zero; at $3 per million input tokens, cost per reply fell from ~$0.54 to ~$0.02.

CONTEXT.md Template

Fill once per task; only bracketed content changes:

ALWAYS FOLLOW (top): [rules that apply to every task, max 15 lines]<br>NEVER LOAD: [content to exclude: old versions, full history, raw logs]<br>RETRIEVE: search [folder / doc / site], return max [3] results, newest first<br>NOTES: read NOTES.md before each step, append one line after, keep failed attempts<br>DELEGATE: any reading task over [20] pages goes to a sub-agent, return [5] lines<br>BUDGET: [N]K tokens per call; if exceeded, compress then continue<br>PINNED (near bottom): [recently changed, must-use facts]<br>GOAL (last line): [the outcome this step must produce]

"Pinned" and "Goal" are deliberately placed at the end because the context tail is the region models miss least.

Diagnose Your Missing Link

When an agent degrades over time, symptoms point to the missing technique:

Follows rules in turn 1, violates by turn 30 → rules stuck in middle; move to top and repeat at end.

Repeats old actions instead of trying new ones → history too long; compress to notes.

Picks wrong tool → too many tools; keep only those the job needs.

Cites stale info → old docs still in context; retrieve latest and pin it.

Reads everything before acting → everything pre-pasted; provide paths and let it search.

Research, coding, writing all in one chat → one window doing five jobs; split reading out.

Six symptoms, six missing links — and a bigger context window fixes none of them.

Mistakes That Neutralize Context Engineering

"It has a 1M-token window, so paste everything" — larger window holds more, doesn't make the model read better (Chroma's data).

Changing the prompt start every call — adding timestamps or reordering tools invalidates cache for everything after. Manus calls cache-hit rate the single most important production metric; cached input costs a fraction of regular input.

Deleting error records — clean history looks tidy but teaches the agent nothing; it retries the same failed approach.

One giant system prompt for all tasks — billing rules mixed into writing context interfere with every answer.

Providing ten examples of the same type — Manus warns: the agent starts mimicking patterns instead of understanding the task.

No per-call budget — context grows until something breaks. Set a hard number; when exceeded, compress.

Where to Start Tonight

Find the agent that runs longer and performs worse.

Count how much it reads per call — most people never look.

Prune the system prompt to only rules that apply every time.

Copy CONTEXT.md, fill the brackets, pin the fact that keeps breaking.

Run the same task again; compare answer quality and bill.

Then look at what you used to paste in and ask how much of it was ever actually used.

P.S. Before you pay for a larger context window, try shrinking the context. It's cheaper and usually works better.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Geminiagent architectureAnthropicLLM context managementtoken budget
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.