Skip to content
Cache Engineering: Agents Directly Manage KV Cache to Cut Inference Costs 27.7%

Cache Engineering: Agents Directly Manage KV Cache to Cut Inference Costs 27.7%

Researchers from Tsinghua and CUHK propose Cache Engineering, a paradigm where agents directly manipulate KV cache via Append, Drop, and Repos operations, avoiding re-prefill overhead; their open-source AKV system reduces average inference cost by 27.7% and improves task success rate by 1.7% across four agent benchmarks.

Machine Heart
Machine Heart
Machine Heart

Context Engineering: Recomputation Overhead

When agents run long-horizon tasks, the history of model inferences and tool calls grows, slowing down the model and increasing costs. Current Context Engineering approaches compress history text by truncation or summarization. However, modifying earlier text invalidates the already-computed KV cache for subsequent tokens, forcing a re-prefill (recomputation) of the remaining context. This re-prefill introduces extra latency and cost, and the compression itself loses historical information, hurting task success rates.

The paper illustrates this with a multi-round web search example: deleting old web pages seems to save tokens, but the KV cache for later reasoning was computed conditioned on those pages. Removing them requires recomputing the cache for the retained portion. On the BrowseComp-Plus benchmark, keeping only the most recent 12 tool-call results actually increased total inference cost to 1.85× that of the full context, revealing that the recomputation overhead can exceed the savings from compression.

Cache Engineering: Direct Cache Management

Instead of modifying text, Cache Engineering retains all historical text and directly shrinks the KV cache that is no longer needed, while reusing the remaining cache. Crucially, the retained KV may still carry information from removed history because later tokens were generated conditioned on that history.

The authors define three primitive cache operations:

Append : Compute new tokens and append their KV to the cache.

Drop : Remove selected KV from future computation, reducing subsequent compute.

Repos (Reposition): Adjust positional encodings of retained cache to avoid exceeding the model's supported context length.

Unlike text operations, these cache operations reuse already-computed states without requiring re-prefill. The agent and its harness can compose these operations at each step to decide which cache entries to keep.

AKV: Inference System Supporting Cache Operations

To realize cache operations, the team built AKV on top of SGLang (https://github.com/sgl-project/sglang). The user submits the full message history plus a cache operation record (e.g., cache.drop and cache.reposition). AKV handles all low-level details:

Determining which cache blocks can be reused, cross-checking text and operation history to avoid mixing states from different sources.

Releasing unneeded cache blocks only when no other request still needs them.

Reconstructing missing cache state in a single prefill when necessary.

A key challenge is that identical text sequences can yield different KV states depending on the sequence of cache operations applied. AKV solves this with an extended radix tree that indexes both the text and the operation history, ensuring correct and efficient cache sharing across divergent trajectories.

Experimental Results: Task Performance and Serving Efficiency

Experiments were run on three models — MiniMax-M2.7 , GPT-OSS-120B , and MiroThinker-1.7-Mini — across four agent benchmarks: BrowseComp-Plus , GAIA-Text , SWE-bench Verified , and xbench-DeepSearch-2510 (covering search and software engineering tasks). Three settings were compared:

Standard (Context Engineering): trigger summarization at a high context threshold.

Text Engineering (Context Engineering): same summarization strategy but with a lower uniform threshold of 48K tokens.

AKV (Cache Engineering): retain reasoning state and KV of the most recent 12 tool responses.

Results (averaged across models and benchmarks):

AKV reduces average inference cost by 27.7% vs. Standard and by 21.9% vs. Text Engineering.

AKV improves average task success rate by 1.7 percentage points vs. Standard.

On 8× A800-80GB GPUs with GPT-OSS-120B, AKV achieves 1.18× output throughput and reduces P95 request latency by about one-third (≈ 1.51× speedup ), because fewer KV entries participate in inference, easing cache capacity pressure and recomputation overhead at high concurrency.

On SWE-bench Verified, AKV reaches higher success rates with less cumulative GPU time.

RSI: Agents Self-Improving Cache Strategies

The fixed "retain last 12 tool responses" policy is just a starting point. The team ran a Recursive Self-Improvement (RSI) experiment: MiniMax-M2.7 searched, evaluated, and iterated cache strategies on 20 BrowseComp-Plus tasks, then validated on 100 held-out tasks. The agent-discovered strategy matched Standard's accuracy while cutting cost a further 7.1% beyond the human-designed policy.

Conclusion and Outlook

The core contribution is Cache Engineering — making KV cache a first-class object that agents and harnesses can directly manage. The open-source AKV system exposes a simple interface for cache operations, letting developers focus on policy design: what to keep, when to drop, and how to adapt to task progress. The rolling-drop policy and RSI experiment are initial explorations; with Cache Engineering and AKV, developers can explore richer KV cache operations and combinations across diverse models, tasks, and budgets.

References

[1] Lindenbauer et al., "The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management," arXiv:2508.21433, 2025.

[2] Wu et al., "ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization," arXiv:2509.13313, 2025.

[3] Zheng et al., "SGLang: Efficient Execution of Structured Language Model Programs," NeurIPS, 2024.

[4] Chen et al., "The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence," arXiv:2605.26494, 2026.

[5] Agarwal et al., "gpt-oss-120b & gpt-oss-20b Model Card," arXiv:2508.10925, 2025.

[6] MiroMind Team, "MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via Verification," arXiv:2603.15726, 2026.

[7] Chen et al., "BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent," arXiv:2508.06600, 2025.

[8] Mialon et al., "GAIA: A Benchmark for General AI Assistants," arXiv:2311.12983, 2023.

[9] OpenAI, "Introducing SWE-bench Verified," 2024.

[10] xbench, "xbench-DeepSearch-2510," Hugging Face dataset, 2025.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

AgentLLM inferenceContext ManagementSGLangKV CacheRSICache EngineeringAKV
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.