Why Bigger 1M‑Token Windows Still Need Careful Context Engineering

Context Engineering Overview

Context Engineering determines which information is loaded into an LLM’s memory before each call, how it is structured, and when it is removed. It complements Prompt Engineering, which focuses on the wording of the instruction itself.

Key Components

System Prompt : static rules (e.g., .cursorrules, AGENTS.md) that define role, goals, constraints, execution flow, and output format.

User Prompt : business data and instructions, often a mix of natural language, fields, and attachments.

Memory : short‑term sliding‑window memory and long‑term stores (vector DBs, KV stores, relational or graph databases) that must be written, updated, forgotten, and recalled.

RAG & Tools : Retrieval‑augmented generation fetches relevant documents; tools provide function calls and results that become part of the context.

Structured Output : JSON schemas or function‑calling signatures that constrain the model’s response.

Token Optimization : summarization, history pruning, and context caching to stay within token budgets while preserving essential information.

Why Larger Context Windows Are Not a Silver Bullet

Even with 1 M‑token windows (e.g., DeepSeek‑V4, GPT‑5.5, Claude Opus 4.6/4.7), performance does not scale linearly with size. Over‑loading the window introduces noise, reduces the signal‑to‑noise ratio, and triggers “Context Rot” – the longer the context, the lower the effective signal. Transformers attend more to the beginning and end of a prompt, so critical middle content can be ignored (“Lost in the Middle”).

Illustrative Example: E‑commerce After‑Sales

When an agent receives only the user’s short message (

MD，我上周买的耳机右耳没声音了，怎么处理？

), it replies with generic clarification questions. If the system pre‑loads order details, warranty status, and historical tickets, the agent can directly propose a replacement, demonstrating that context quality outweighs sheer quantity.

Evaluating Context Engineering

Intuition is insufficient; track five metric groups on a small evaluation set (20‑50 real task traces) and change one variable at a time.

Task Success Rate : goal completion, need for manual rescue, reproducibility.

Tool Quality : wrong tool selection, missing parameters, duplicate calls, safety interceptions.

Context Cost : input/output token count, cache hit rate, information retention after compression.

Latency : first‑token latency, end‑to‑end time, tool wait time, p95/p99 response.

Result Quality : hallucination rate, citation accuracy, summary loss, key‑field omission.

Runtime Context Loading Strategies

Pre‑retrieval vs. Just‑in‑Time (JIT)

Pre‑retrieval fetches all seemingly relevant documents before the LLM call – suitable for simple Q&A but brittle for complex agents that discover new clues during execution. JIT loading defers fetching until the agent actually needs the data, using lightweight references (file paths, DB queries) and tools such as head, tail, grep. Anthropic calls this “Progressive Disclosure”.

Hybrid Approach

Most production systems combine both: static knowledge is pre‑retrieved, while dynamic information is loaded JIT. Choose based on task characteristics – codebase analysis or fault diagnosis favor JIT; stable document review favors pre‑retrieval.

Managing Long‑Running Tasks

Compaction

When the window fills, summarize past messages with the LLM and start a new window, retaining high‑level decisions, unresolved bugs, and key details. Anthropic’s Claude Code example keeps the five most recent files after compression.

Structured Note‑Taking

Agents write progress notes to external files (e.g., NOTES.md) and reload them after a context reset, ensuring continuity across long interactions.

Sub‑Agents

Split a large task into specialized sub‑agents that each explore massive context (tens of thousands of tokens) and return a concise 1‑2 K‑token summary to the master agent, keeping the master’s context clean.

Practical Context Assembly Pipeline

# Input: user_task, session_state, business_context
constraints = load_system_constraints()
goal = extract_current_goal(user_task, session_state)
evidence = retrieve_rag(goal, business_context)
memory = recall_memory(goal, session_state)
tools = select_tools(goal, evidence, memory)
history = compact_history(session_state.messages)
context = rank([constraints, goal, evidence, memory, tools, history])
context = fit_token_budget(context)
# Output: messages, tool_schema, metadata

Two critical steps: rank decides the ordering of information. fit_token_budget determines what stays raw, what is summarized, and what is kept as a reference.

Building the Foundations

Static Rules (System Prompt)

Write a concise, structured System Prompt in Markdown, separating role, constraints, execution flow, and output format. Example for a backend fault‑diagnosis agent:

## Role
You are a backend service fault‑diagnosis expert.
## Constraints
- Call only necessary tools.
- Stop searching once key evidence is found.
- Prefer real‑time data over historical inference.
## Execution Flow
1. Check monitoring metrics.
2. Query logs for the time window.
3. Trace upstream dependencies if anomalies appear.
4. Output structured report.
## Output Format
JSON with fields: incident_summary, root_cause, evidence, recommendation

Store these files (e.g., .cursorrules, AGENTS.md) for team‑wide consistency.

Tool Descriptions

Good tool schemas answer two questions: when to invoke and when not to. Overly broad tools cause hesitation and parameter noise. Each tool should do one thing and include clear input examples.

Dynamic Context (RAG, Memory, Tool Results)

After retrieval, short‑term memory is managed with a sliding window; long‑term facts reside in external stores. Tool results should be trimmed or summarized, but retain raw identifiers (trace IDs, timestamps) for debugging.

Few‑Shot Examples

Provide 3‑5 diverse canonical examples rather than dozens of edge cases. This teaches the model the mapping between situation and strategy without overfitting to surface forms.

Token Budget Prioritization

Within a single call, prioritize high‑importance items (system constraints, current goal, safety boundaries) over low‑importance history, which can be AI‑summarized.

Low (foldable) : early dialogue history – AI summarization.

Medium (compressible) : RAG background, old tool results – second‑pass trimming, keep core paragraphs.

High (fixed) : system constraints, current goal, safety limits – fixed high‑priority slot.

Stage‑specific : tool schemas, few‑shot examples needed now – load on demand, unload after use.

Tooling Landscape

Orchestration frameworks: LangChain, LangGraph

Data frameworks: LlamaIndex for RAG pipelines

Vector stores: Pinecone, Weaviate, Chroma, Qdrant

Communication protocol: MCP (JSON‑RPC 2.0 based)

Memory products: Mem0, LETTA, ZEP

Practical Recommendations

Prioritize signal‑to‑noise over raw token count; aim for a context utilization of roughly 40‑60 %. Start with the simplest working pipeline (clear System Prompt, well‑defined tools, basic RAG). Add compaction, note‑taking, or sub‑agents only when the task truly demands them. A well‑engineered context can enable a modest model to solve complex problems, while a noisy context will cripple even the most powerful model.