BestHub
Sign in
Tagged articles

Prefix Caching

6 articles · Page 1 of 1
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Oct 13, 2026 · Artificial Intelligence

LightRSI TokenPilot: Slash LLM Agent Context Costs Up to 87% with Smart Cleanup

Zhejiang University's LightRSI project introduces TokenPilot, an EMNLP 2026 accepted system for cache-efficient context management in LLM agents, combining prefix cache stabilization, tool output summarization, and task-level context cleaning to reduce inference costs by up to 87% on Claw-Eval while integrating with six coding agents and providing a visualization dashboard.

EMNLP 2026LLM agentsLightRSI
0 likes · 13 min read
LightRSI TokenPilot: Slash LLM Agent Context Costs Up to 87% with Smart Cleanup
Architecture Development Notes
Architecture Development Notes
Sep 4, 2026 · Artificial Intelligence

75% Cheaper Cache Reads: Why Long-Running Agent Costs Now Depend on Prefix Stability

Anthropic's Fable 5.1 reduces cache read pricing from $1 to $0.25 per million tokens, shifting long-running agent cost bottlenecks from output to repeated stable prefix reads, making prefix stability, cache breakpoint placement, TTL tuning, and hit-rate observability critical architectural levers for cost control.

AI Agent ArchitectureAnthropicFable 5.1
0 likes · 15 min read
75% Cheaper Cache Reads: Why Long-Running Agent Costs Now Depend on Prefix Stability
James' Growth Diary
James' Growth Diary
Jun 25, 2026 · Artificial Intelligence

Why Compression Isn’t Truncation: Hermes’s Structured Summaries Keep Prefix Cache Hits

The article explains how Hermes Agent avoids the pitfalls of naive sliding‑window truncation—such as orphaned tool calls and broken KV‑cache—by using a three‑segment protection scheme, cheap tool‑result pre‑pruning, and a structured, reference‑only summary that dramatically reduces tokens while preserving and even improving prefix cache hit rates.

Hermes AgentLLMPrefix Caching
0 likes · 17 min read
Why Compression Isn’t Truncation: Hermes’s Structured Summaries Keep Prefix Cache Hits
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 7, 2026 · Artificial Intelligence

Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained

The article explains why LLM inference is memory‑bound, introduces vLLM’s three core optimizations—Continuous Batching, PagedAttention, and Prefix Caching—shows how to launch a vLLM server, run Python code to benchmark performance, and examines KV‑Cache memory usage with concrete numbers.

KV cacheLLM inferencePagedAttention
0 likes · 11 min read
Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained
SuanNi
SuanNi
May 14, 2026 · Artificial Intelligence

How a Pure‑Software Framework Boosts On‑Device AI Agents by 1.6×

KAIST researchers introduced Agent‑X, a pure‑software acceleration framework that eliminates prefill and decode bottlenecks on mobile devices, achieving a 1.61× end‑to‑end speedup for on‑device AI agents without any loss in task accuracy.

Agent-XKAISTPrefix Caching
0 likes · 9 min read
How a Pure‑Software Framework Boosts On‑Device AI Agents by 1.6×