BestHub
Sign in
Tagged articles

prefix caching

4 articles · Page 1 of 1
James' Growth Diary
James' Growth Diary
Jun 25, 2026 · Artificial Intelligence

Why Compression Isn’t Truncation: Hermes’s Structured Summaries Keep Prefix Cache Hits

The article explains how Hermes Agent avoids the pitfalls of naive sliding‑window truncation—such as orphaned tool calls and broken KV‑cache—by using a three‑segment protection scheme, cheap tool‑result pre‑pruning, and a structured, reference‑only summary that dramatically reduces tokens while preserving and even improving prefix cache hit rates.

Context CompressionHermes AgentLLM
0 likes · 17 min read
Why Compression Isn’t Truncation: Hermes’s Structured Summaries Keep Prefix Cache Hits
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 7, 2026 · Artificial Intelligence

Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained

The article explains why LLM inference is memory‑bound, introduces vLLM’s three core optimizations—Continuous Batching, PagedAttention, and Prefix Caching—shows how to launch a vLLM server, run Python code to benchmark performance, and examines KV‑Cache memory usage with concrete numbers.

Continuous BatchingKV cachePagedAttention
0 likes · 11 min read
Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained
SuanNi
SuanNi
May 14, 2026 · Artificial Intelligence

How a Pure‑Software Framework Boosts On‑Device AI Agents by 1.6×

KAIST researchers introduced Agent‑X, a pure‑software acceleration framework that eliminates prefill and decode bottlenecks on mobile devices, achieving a 1.61× end‑to‑end speedup for on‑device AI agents without any loss in task accuracy.

Agent-XKAISTdecode bottleneck
0 likes · 9 min read
How a Pure‑Software Framework Boosts On‑Device AI Agents by 1.6×