How AutoContextMemory Cuts LLM Costs by 70% in Long Conversations

Problem

In multi‑turn conversations the token count grows linearly. By the 100th turn an API call may need ~100 000 tokens, causing linear cost increase, slower inference and hitting the model’s max‑token limit.

Solution – AutoContextMemory

AutoContextMemory is a component of the AgentScope Java framework that automatically compresses, offloads and summarizes dialogue history. It reduces token usage while preserving essential information.

Core features

Automatic compression & summarization : When message count or token ratio exceeds configurable thresholds, six progressive strategies are applied, including LLM‑based summarization.

Content offloading & traceability : Large messages are moved to external storage identified by a UUID; the original uncompressed history remains for full traceability.

Performance gains : Benchmarks show up to 70 % token reduction and ~60 % faster response times without degrading decision quality.

Architecture

AutoContextMemory uses a multi‑storage design:

Working memory : Stores compressed messages that are fed to the model.

Raw memory : Append‑only store of the full, uncompressed history.

Offload storage : Holds messages moved out of working memory, retrievable by UUID.

Compression event storage : Records details of each compression operation for analysis.

All storages support persistence and can be combined with SessionManager for cross‑session context.

Compression strategies

Six built‑in strategies are evaluated in order when thresholds are met:

Compress consecutive tool‑call messages (≥6) using LLM summarization.

Offload large messages with protection (keep latest assistant reply and last N messages).

Offload large messages without protection (keep only latest assistant reply).

Summarize historical user‑assistant rounds.

Summarize large messages in the current round.

Compress all messages in the current round as a fallback.

Trigger conditions are either a message‑count threshold ( msgThreshold) or a token‑ratio threshold ( tokenRatio).

Usage example

Add the following Maven dependencies to pom.xml:

<dependencies>
  <dependency>
    <groupId>io.agentscope</groupId>
    <artifactId>agentscope-core</artifactId>
    <version>1.0.2</version>
  </dependency>
  <dependency>
    <groupId>io.agentscope</groupId>
    <artifactId>agentscope-extensions-autocontext-memory</artifactId>
    <version>1.0.2</version>
  </dependency>
</dependencies>

Configure and register the memory:

import io.agentscope.core.ReActAgent;
import io.agentscope.core.memory.autocontext.AutoContextConfig;
import io.agentscope.core.memory.autocontext.AutoContextMemory;
import io.agentscope.core.memory.autocontext.ContextOffloadTool;
import io.agentscope.core.tool.Toolkit;

// Build config
AutoContextConfig config = AutoContextConfig.builder().build();

// Create memory instance
AutoContextMemory memory = new AutoContextMemory(config, model);

// Register offload tool
Toolkit toolkit = new Toolkit();
toolkit.registerTool(new ContextOffloadTool(memory));

// Build agent
ReActAgent agent = ReActAgent.builder()
    .name("Assistant")
    .model(model)
    .memory(memory)
    .toolkit(toolkit)
    .enablePlan()
    .build();

Performance evaluation

A 20 000‑word code‑review task on Nacos server configuration code was run with and without AutoContextMemory (five runs each). Results:

Token consumption : average reduced from ~6.2 M to ~2.1 M (≈68 % reduction).

Response time : total runtime dropped from ~1.5 h to ~30 min (≈58 % reduction).

Analysis & optimization tools

AutoContextMemory records compression events, allowing developers to inspect working‑memory vs. raw‑memory sizes, offloaded entry counts and event statistics. Key parameters that can be tuned include msgThreshold, tokenRatio, lastKeep, largePayloadThreshold, and minConsecutiveToolMessages. Typical optimization scenarios are described in the documentation.

Resources

GitHub repository: https://github.com/agentscope-ai/agentscope-java