Agent Cache Engineering: Tsinghua & CUHK's AKV Cuts Inference Cost 27.7% via RSI

Problem: KV Cache Growth in Multi-Turn Agent Interactions

As agents operate over multiple turns, the key-value (KV) cache expands with each interaction, increasing storage, memory access, and inference costs. Traditional context engineering compresses history by truncation or summarization, but modifying the historical prefix invalidates all subsequent KV cache, forcing a full re-prefill. The cost saved by compression is often lost in re-prefill overhead, and aggressive compression degrades task success rates.

Cache Engineering Paradigm

The core idea is to let the agent manage its own KV cache directly, discarding unneeded history and relocating retained key-values without rewriting the logical text history, thereby avoiding re-prefill. The researchers decouple the logical trajectory (full prompt, reasoning messages, tool responses) from the active cache (KV states actually participating in attention). This decoupling enables cache resizing while keeping the text history intact.

Three Primitive Cache Operations

Append : Add new token KV states (from prompt, tool response, or model generation) to the active cache, maintaining autoregressive inference.

Drop : Remove KV states at specified token indices from the active cache so they no longer participate in attention. The remaining cache and their positional encodings stay unchanged, and subsequent appends use the same position indices, so retained KV need not be recomputed.

Repos (Position Remapping) : Re-index the positional encodings of retained cache to a contiguous range starting at 0, decoupling position encodings from logical indices and cache size. For RoPE, keys are updated via K′ = R(q)R(p)⁻¹K while values remain unchanged.

These operations form a programmable cache operation sequence, allowing dynamic adjustment based on message role, timeliness, and task requirements. Critically, they affect only the requesting request's attention and do not alter the output distribution of other requests sharing the same prefix.

AKV Inference Serving System

AKV adopts a stateless interface, stateful service architecture with three key mechanisms:

Extended Radix Tree for Accurate Cache Reuse

The system extends the traditional radix tree with Drop nodes and Repos nodes that record where cache operations occurred and their effects. Cache matching considers both token sequence and operation history, distinguishing KV states generated under different historical conditions to prevent incorrect reuse.

Cache-Operation-Aware Eviction

When a KV entry is no longer needed by any in-flight computation or recovery process, its physical storage is released. Among evictable entries, the system prioritizes dropping already-dropped caches (since completed-request caches are more likely to be reused next round) while preserving logical tree nodes to maintain correctness of future cache matching and state recovery. Different positional-encoding versions created by Repos are managed with separate lifecycles to avoid interference between concurrent requests.

Segment-Batch Attention for Efficient Recovery

When cache is evicted due to memory pressure, AKV does not replay historical operations step by step. Instead, it partitions attention computation into segments at Drop and Repos boundaries. Each segment runs attention with the visible cache and positional encodings from its original execution. Dropped cache remains available for earlier segments that depended on it; for Repos, the system constructs temporary KV versions via key rotation and value copying to restore the positional encodings of that phase. The feed-forward network (FFN) processes the full sequence in a single batch, recovering required state in one prefill and reducing redundant computation and model weight reads.

Tool-Response Rolling Drop Strategy

Attention analysis on 10 BCP requests revealed that earlier tokens receive lower attention weights, and at similar historical positions, tool responses receive lower attention than agent replies. Based on this, the authors designed the Tool-Response Rolling Drop strategy: retain all agent reasoning message KV cache, keep only the most recent 12 tool response KV states, and progressively drop older tool response caches. When the position index reaches a preset threshold aligned with each model's summarization threshold, a Repos operation re-encodes positions, balancing long-term information retention, cache size control, and inference efficiency.

Recursive Self-Improvement (RSI) for Cache Policy Optimization

Beyond hand-designed policies, the team explored whether the model could optimize its own cache strategy. They used MiniMax-M2.7 to iteratively refine cache policies on 20 BCP tasks, then evaluated on 100 held-out tasks. By iteration 3 and 6, the learned policies reduced per-task cost by 15.6% and 17.8% respectively, while accuracy remained on par with the Standard baseline, demonstrating that RSI can discover more cost-effective cache management strategies than human design.

Benchmark Results

Compared to common threshold-triggered context compression, AKV achieves:

Average 27.7% reduction in inference cost

Task score improvement of 1.7 percentage points

Maximum output throughput increase of 1.18×

Maximum P95 latency speedup of 1.51×

Detailed results are shown in Table 1 (model scores and costs) and Table 2 (GPT-OSS-120B serving performance under varying concurrency).

AKV Installation Tutorial

1. Install Rust and Cargo (if not present)

curl -o install-rust.sh https://sh.rustup.rs
sh install-rust.sh -y
source "$HOME/.cargo/env"

2. Create AKV Environment and Install Agent & System

git clone https://github.com/thu-nics/AKV.git
cd AKV
uv venv --python 3.12
source .venv/bin/activate
uv pip install -e .

3. Launch AKV Server

python -m akv.system.launch_server \
  --model-path /path/to/gpt-oss-120b \
  --reasoning-parser gpt-oss \
  --tool-call-parser gpt-oss \
  --tp-size 2 \
  --port 8000