BestHub
Sign in
Tagged articles

prefix caching

5 articles · Page 1 of 1
Architecture Development Notes
Architecture Development Notes
Sep 4, 2026 · Artificial Intelligence

75% Cheaper Cache Reads: Why Long-Running Agent Costs Now Depend on Prefix Stability

Anthropic's Fable 5.1 reduces cache read pricing from $1 to $0.25 per million tokens, shifting long-running agent cost bottlenecks from output to repeated stable prefix reads, making prefix stability, cache breakpoint placement, TTL tuning, and hit-rate observability critical architectural levers for cost control.

AI Agent ArchitectureAnthropicContext Management
0 likes · 15 min read
75% Cheaper Cache Reads: Why Long-Running Agent Costs Now Depend on Prefix Stability
James' Growth Diary
James' Growth Diary
Jun 25, 2026 · Artificial Intelligence

Why Compression Isn’t Truncation: Hermes’s Structured Summaries Keep Prefix Cache Hits

The article explains how Hermes Agent avoids the pitfalls of naive sliding‑window truncation—such as orphaned tool calls and broken KV‑cache—by using a three‑segment protection scheme, cheap tool‑result pre‑pruning, and a structured, reference‑only summary that dramatically reduces tokens while preserving and even improving prefix cache hit rates.

Context CompressionHermes AgentLLM
0 likes · 17 min read
Why Compression Isn’t Truncation: Hermes’s Structured Summaries Keep Prefix Cache Hits
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 7, 2026 · Artificial Intelligence

Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained

The article explains why LLM inference is memory‑bound, introduces vLLM’s three core optimizations—Continuous Batching, PagedAttention, and Prefix Caching—shows how to launch a vLLM server, run Python code to benchmark performance, and examines KV‑Cache memory usage with concrete numbers.

KV CacheLLM inferencePagedAttention
0 likes · 11 min read
Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained
SuanNi
SuanNi
May 14, 2026 · Artificial Intelligence

How a Pure‑Software Framework Boosts On‑Device AI Agents by 1.6×

KAIST researchers introduced Agent‑X, a pure‑software acceleration framework that eliminates prefill and decode bottlenecks on mobile devices, achieving a 1.61× end‑to‑end speedup for on‑device AI agents without any loss in task accuracy.

Agent-XKAISTdecode bottleneck
0 likes · 9 min read
How a Pure‑Software Framework Boosts On‑Device AI Agents by 1.6×