Deep Agents Context Management: How SummarizationMiddleware Auto-Compresses History

1. Two Middlewares for Context Management

The summarization.py module uses an LLM to generate summaries, offloads full history to backend storage, and controls context window size. It provides two middlewares: SummarizationMiddleware — automatically compresses context during model calls. SummarizationToolMiddleware — exposes a compact_conversation tool that agents or users can trigger manually.

Both share the same compression logic; only the trigger mechanism differs. The article focuses on the automatic flow first.

2. SummarizationMiddleware

2.1 compute_summarization_defaults

Compression involves two parameters: trigger threshold and keep ratio. compute_summarization_defaults sets them based on the model profile.

If the model profile defines max_input_tokens:

{
  "trigger": ("fraction", 0.85),
  "keep": ("fraction", 0.10),
  "truncate_args_settings": {
    "trigger": ("fraction", 0.85),
    "keep": ("fraction", 0.10),
  },
}

Trigger compression when context exceeds 85% of the window.

Retain 10% of the context.

If no profile exists, defaults are:

{
  "trigger": ("tokens", 170000),
  "keep": ("messages", 6),
  "truncate_args_settings": {
    "trigger": ("messages", 20),
    "keep": ("messages", 20),
  }
}

Trigger when token count exceeds 170,000.

Retain the 6 most recent messages.

2.2 SummarizationMiddleware Initialization

LCSummarizationMiddleware

contains the core compression logic. SummarizationMiddleware extends it with: history_path_prefix — path prefix for storing conversation history. truncate_args_settings — configuration for truncating large tool arguments.

2.3 SummarizationMiddleware Hook: wrap_model_call

The wrap_model_call hook runs before each model invocation to decide whether summarization is needed, and after to update state. The pseudocode below captures the full flow:

def wrap_model_call(request, handler):
    # 0) Rebuild effective messages from prior summarization events
    effective_messages = get_effective_messages(request)

    # 1) Lightweight truncation of large tool args in older messages
    truncated_messages, _ = truncate_args(
        effective_messages,
        system_message=request.system_message,
        tools=request.tools,
    )

    # 2) Count tokens and decide whether to summarize
    total_tokens = token_counter_with_compat(counted_messages=[system_message + truncated_messages])
    should_summarize = should_summarize_fn(truncated_messages, total_tokens)

    # 2.1 If not needed, attempt normal call
    if not should_summarize:
        try:
            return handler(request.override(messages=truncated_messages))
        except ContextOverflowError:
            # Fall back to summarization path if normal call overflows
            pass

    # 3) Determine cutoff index for splitting history
    cutoff = determine_cutoff_index(truncated_messages)
    if cutoff <= 0:
        # Cannot compress; fall back to normal call
        return handler(request.override(messages=truncated_messages))

    # 4) Partition messages into to_summarize and preserved
    to_summarize, preserved = partition_messages(truncated_messages, cutoff)

    # 5) Offload to_summarize to backend (non-blocking)
    backend = get_backend(request.state, request.runtime)
    file_path = offload_to_backend(backend, to_summarize)
    if file_path is None:
        log_error_and_warn_user()

    # 6) Generate summary and build new messages with optional file_path hint
    summary = create_summary(to_summarize)
    new_messages = build_new_messages_with_path(summary, file_path)

    # 7) Compute absolute state cutoff (supports chained summarization)
    prev_event = request.state.get("_summarization_event")
    state_cutoff = compute_state_cutoff(prev_event, cutoff)

    # 8) Call model with summary + preserved messages
    response = handler(request.override(messages=[*new_messages, *preserved]))

    # 9) Return extended response with new summarization_event in state
    new_event = {
        "cutoff_index": state_cutoff,
        "summary_message": new_messages[0],
        "file_path": file_path,
    }
    return ExtendedModelResponse(response, Command(update={"_summarization_event": new_event}))

Rebuilding Effective Messages

_get_effective_messages()

reads request.state["_summarization_event"] and reconstructs the input as the summary message plus all messages after the stored cutoff index:

result: list[AnyMessage] = [summary_msg]
result.extend(messages[cutoff_idx:])

Truncating Tool Arguments

_truncate_args()

is a lightweight optimization that truncates large string arguments inside AIMessage.tool_calls for older messages, reducing token usage and often avoiding the need for full summarization.

Attempting the Model Call

_should_summarize

evaluates the threshold. If not met, the middleware attempts a normal model call. If a ContextOverflowError occurs, it falls back to the summarization path.

Cutoff Index Determines Compressibility

_determine_cutoff_index()

sets the compression boundary: cutoff_index <= 0: no compressible range; use truncated messages directly, no summarization event written. cutoff_index > 0: proceed to partition and summary replacement.

Offload Before Summarize for Traceability

Messages to be summarized are first persisted via _offload_to_backend(). Failure does not block the main flow; it logs an error and warning, sets file_path=None, and continues summary generation.

Summary Message Includes External History Reference

_build_new_messages_with_path()

embeds the backend file path in the summary message when available, enabling later retrieval of full details. Without a path, it degrades to plain summary text. Example template:

You are in the middle of a conversation that has been summarized.

The full conversation history has been saved to {file_path} should you need to refer back to it for details.

A condensed summary follows:

<summary>
{summary}
</summary>

State Cutoff Index Maps Back to Original Message Array

_compute_state_cutoff(previous_event, cutoff_index)

translates the effective-list cutoff back to the original state message index using prior_cutoff + effective_cutoff - 1. The -1 offsets the summary placeholder at the head of the effective list.

2.4 Three Message Views

The system maintains three distinct views of the conversation:

State Messages (raw view)
    │
    │ Never deleted, only appended
    │ [M0, M1, M2, M3, ..., M20, M21, ...]
    │
    ├─> Effective Messages (reconstructed view)
    │     │
    │     │ Rebuilt from _summarization_event
    │     │ _get_effective_messages(event)
    │     │
    │     └─> [Summary_N, M_k, M_{k+1}, ..., M_latest]
    │           ↑                ↑
    │      compressed         from last cutoff
    │      summary
    │
    └─> Model Messages (sent to model)
          │
          │ Subset of Effective Messages
          │ Processed by _truncate_args()
          │
          └─> [Summary_N, M_k, ..., M_latest]
                  (tool args may be truncated)

Backend File Structure

Each conversation thread stores a Markdown file with timestamped sections:

📁 /conversation_history/{thread_id}.md
├── ## Summarized at 2025-01-15T10:00:00
│   └── Messages [0-5] full content
│
├── ## Summarized at 2025-01-15T11:30:00
│   └── Messages [6-13] full content (excludes Summary#1)
│
└── ## Summarized at 2025-01-15T13:00:00
    └── Messages [14-22] full content (excludes Summary#2)

State _summarization_event

{
  "cutoff_index": 22,          ← index in original messages array
  "summary_message": HumanMessage("..."),  ← current summary
  "file_path": "..."           ← backend file path
}

3. SummarizationToolMiddleware

This middleware exposes a compact_conversation tool implemented in _run_compact:

def _run_compact(runtime):
    # 1) Initialization
    messages = get messages from state
    event = get summarization event from state
    effective_messages = apply event to messages

    # 2) Eligibility check
    if not eligible_for_compaction(effective_messages):
        return "No compaction needed"

    # 3) Determine cutoff
    cutoff = compute cutoff index
    if cutoff == 0:
        return "No compaction needed"

    # 4) Execute compaction
    try:
        old_messages, new_messages = split messages at cutoff
        summary = create_summary(old_messages)
        file_path = offload_to_backend(old_messages)
        return build_compaction_result(summary, file_path)
    except Exception:
        return error message

Compression logic mirrors the automatic flow, with two key differences:

Trigger threshold : _is_eligible_for_compaction uses 50% of the automatic trigger threshold.

Error handling : Exceptions are caught and returned as ToolMessage errors without interrupting execution; if ineligible, returns a "nothing to compact" message.