10 Open‑Source Tools Cutting AI Agent Costs Ten‑Fold: Prompt Compression, Memory Management, Model Routing

Context is the primary driver of AI‑agent cost

Teams often blame high model fees, but the dominant expense is the large amount of context (prompt, chat history, codebase) sent to the model on each call. Supplying unnecessary tokens is analogous to moving an entire house when only a vase needs transport, leading to token waste and inflated bills.

Prompt compression with LLMLingua

LLMLingua (https://github.com/microsoft/LLMLingua) removes filler words from system prompts, keeping only high‑information tokens. A typical “employee‑handbook” prompt can exceed 2,000 tokens. Example:

请仔细阅读以下用户问题，并根据你的知识提供准确简洁的回答。

After compression: 读问题，准确回答。 Saving ~500 tokens per request translates to five million tokens per day for a service that makes 10,000 API calls daily, roughly the cost of a meal per day.

Long‑term memory with mem0

mem0 (https://github.com/mem0ai/mem0) treats chat logs as a diary and extracts salient facts after each turn using a small model. For a conversation containing 20 k tokens, mem0 reduces it to a few dozen tokens such as:

用户操作系统：macOS

技术栈：Next.js

包管理工具：pnpm

This compression cuts request size by 70‑80% compared with sending the full history.

Dynamic model routing with LiteLLM

LiteLLM (https://github.com/BerriAI/litellm) dispatches tasks to the cheapest adequate model based on difficulty:

Simple extraction → inexpensive small model

Contract summarization → mid‑tier model

Complex code debugging → top‑tier model

The system also provides automatic downgrade if a model fails, preventing total service outage.

Retrieval‑augmented generation (RAG) with LlamaIndex and Chroma

LlamaIndex (https://github.com/run-llama/llama_index) and Chroma (https://github.com/chroma-core/chroma) implement a RAG pipeline that limits the text sent to the LLM to the most relevant passages. The process consists of five steps:

User asks a question.

System encodes the question into a semantic vector.

Chroma searches the vector against the document store and returns the top‑few matching snippets.

The retrieved snippets are combined with the original query.

The LLM generates the answer from this concise context.

By retrieving only a few relevant paragraphs instead of feeding an entire knowledge base, token consumption is dramatically reduced.

Paging memory with Letta

Letta (https://github.com/letta-ai/letta) applies operating‑system‑style virtual memory to LLM context. Recent turns are kept in a fast cache, older turns are compressed into summaries, and very old turns are stored as key‑memory cards. In a 100‑turn conversation (~20 k tokens), Letta retains full recent context and compresses the rest to a few hundred tokens, achieving an 80% reduction in per‑request token count.

Structured output with Guidance

Guidance (https://github.com/guidance-ai/guidance) constrains each generation step, eliminating polite preambles and malformed JSON. Example constraint for an age field: gen("age", regex="[0-9]+") The model can only emit digits, guaranteeing correctly formatted output and cutting retry‑related token waste by up to 50%.

Code‑repository map with Aider

Aider (https://github.com/Aider-AI/aider) scans a code repository, builds a dependency graph, and loads only files relevant to the user’s question. For a request about main.py, Aider includes utils.py (a direct dependency) but excludes unrelated files such as test_auth.py. This turns a multi‑million‑token codebase into a focused search problem, avoiding the need to feed the entire repository to the model.

Token budgeting tools tiktoken and ttok

tiktoken (https://github.com/openai/tiktoken) provides model‑specific token counting, allowing developers to estimate request cost before calling the API. ttok (https://github.com/simonw/ttok) is a CLI wrapper that can truncate input to a token limit (e.g., 4 k tokens) directly from the command line, simplifying token‑budget enforcement in scripts.

Context‑engineering pillars

The techniques above form five pillars of context engineering:

Prompt compression

Memory architecture

RAG pipeline

Model routing

Token economics (budgeting and monitoring)

Together they reduce the amount of text the model reads, delivering substantial cost savings regardless of the underlying model price.