BestHub
Sign in
Tagged articles

KV cache

85 articles · Page 1 of 1
Machine Heart
Machine Heart
Jul 21, 2026 · Artificial Intelligence

Can Multi-Agent Systems Be Built Like LEGO? Introducing Agent Primitives for Modular Reuse

The paper proposes Agent Primitives—reusable latent building blocks for multi‑agent systems—that replace hand‑crafted pipelines with modular collaboration patterns, communicate via KV‑Cache to avoid natural‑language bottlenecks, and demonstrate significant accuracy, efficiency, and stability gains across diverse tasks and LLM backbones.

AI researchKV cacheLLM communication
0 likes · 12 min read
Can Multi-Agent Systems Be Built Like LEGO? Introducing Agent Primitives for Modular Reuse
MaGe Linux Operations
MaGe Linux Operations
Jul 19, 2026 · Operations

How to Deploy vLLM for an OpenAI‑Compatible Inference Service

This guide walks through deploying vLLM on Linux with NVIDIA GPUs and Docker Compose, covering service boundaries, host and container checks, model directory validation, image and parameter verification, minimal startup, Compose configuration, API testing, concurrency tuning, multi‑GPU troubleshooting, Nginx exposure, upgrade/rollback procedures, and daily health checks.

Docker ComposeGPUKV cache
0 likes · 18 min read
How to Deploy vLLM for an OpenAI‑Compatible Inference Service
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
Jul 16, 2026 · Artificial Intelligence

Cut First‑Token Latency by 3.25×: Introducing HYPIC for Position‑Independent Caching in Hybrid‑Attention LLMs

HYPIC combines position‑independent caching with hybrid‑attention LLMs, reducing first‑token latency by 3.25× and sustaining QPS by 1.66× while keeping quality loss under 2 points, through a segment‑cumulative transition operator, a seam‑window fix for full‑attention layers, and segment‑parallel execution.

HYPICHybrid AttentionKV cache
0 likes · 13 min read
Cut First‑Token Latency by 3.25×: Introducing HYPIC for Position‑Independent Caching in Hybrid‑Attention LLMs
ThinkingAgent
ThinkingAgent
Jul 5, 2026 · Artificial Intelligence

Building L1 AI Infra: Model Gateways, Smart Routing, and High‑Performance Inference Engines

The article presents a comprehensive, production‑ready guide for the L1 layer of AI infrastructure, detailing how model gateways unify calls, intelligent routing selects the optimal model, inference engines maximize GPU throughput, and quantization and KV‑Cache techniques dramatically cut costs while maintaining performance.

AI infrastructureKV cacheLLM routing
0 likes · 25 min read
Building L1 AI Infra: Model Gateways, Smart Routing, and High‑Performance Inference Engines
Machine Heart
Machine Heart
Jul 3, 2026 · Artificial Intelligence

Avoiding Pitfalls in Heterogeneous Token Factories: Industry‑Level Design Practices for Cross‑Hardware LLM Inference

The article analyzes a recent multi‑institution paper that maps the design space of heterogeneous Prefill‑Decode LLM inference, identifies three core boundary decisions, presents nine deployment best practices, and validates them with a production token‑factory case on MuXi C600 and NVIDIA Hopper GPUs.

KV cacheLLMdeployment best practices
0 likes · 11 min read
Avoiding Pitfalls in Heterogeneous Token Factories: Industry‑Level Design Practices for Cross‑Hardware LLM Inference
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Jul 1, 2026 · Artificial Intelligence

SQL‑Driven Text Classification with Hologres AI Function: Prompt Design to KV‑Cache Tuning

This article demonstrates how Hologres AI Function enables end‑to‑end text classification directly in the database using a single SQL call, covering data preparation, prompt engineering, batch inference, accuracy evaluation (up to 95%), and cost analysis with KV‑Cache optimization that reduces token charges to as low as 0.11 CNY for 200 reviews.

AI FunctionCost OptimizationHologres
0 likes · 12 min read
SQL‑Driven Text Classification with Hologres AI Function: Prompt Design to KV‑Cache Tuning
AI Tech Publishing
AI Tech Publishing
Jun 29, 2026 · Artificial Intelligence

Understanding the Full LLM Inference Pipeline: From Tokenization to Streaming Output

The article walks through every stage of LLM inference—from tokenization and embedding, through transformer layers, prefill (compute‑bound) and decode (memory‑bound) phases, KV‑cache management, attention redesign, quantization, and serving infrastructure—explaining how each step consumes time and resources and how to identify bottlenecks.

Attention optimizationKV cacheQuantization
0 likes · 14 min read
Understanding the Full LLM Inference Pipeline: From Tokenization to Streaming Output
Raymond Ops
Raymond Ops
Jun 28, 2026 · Operations

Why Large‑Model Services Keep Running Out of GPU Memory: An Ops View from KV Cache to Concurrency

The article explains why large‑model inference services frequently hit GPU memory limits, breaks down static vs. dynamic memory consumption, shows how KV‑Cache, request length, and concurrency amplify usage, and provides a step‑by‑step troubleshooting and mitigation workflow for production environments.

Capacity PlanningGPU memoryKV cache
0 likes · 26 min read
Why Large‑Model Services Keep Running Out of GPU Memory: An Ops View from KV Cache to Concurrency
DataFunSummit
DataFunSummit
Jun 21, 2026 · Artificial Intelligence

Unified Scheduling Optimization for xLLM in Complex Business Scenarios

This article analyzes how the xLLM open‑source LLM inference engine tackles the coexistence of multiple priority levels and strict SLO latency targets by introducing a dynamic, SLO‑aware batch scheduler and a PD‑separation architecture that improve throughput and SLO satisfaction across diverse workloads.

Hierarchical Block ManagerKV cachePD Separation
0 likes · 13 min read
Unified Scheduling Optimization for xLLM in Complex Business Scenarios
DataFunTalk
DataFunTalk
Jun 19, 2026 · Artificial Intelligence

How NVIDIA Dynamo Boosts Multi‑Node Distributed Inference MFU for Agentic AI

The article explains how NVIDIA Dynamo tackles the production bottlenecks of Agentic AI by using KV‑Cache‑aware routing, a three‑stage multimodal inference architecture, and intelligent cache scheduling on Kubernetes to improve multi‑node throughput (MFU) while maintaining latency SLAs.

Distributed InferenceKV cacheKubernetes
0 likes · 3 min read
How NVIDIA Dynamo Boosts Multi‑Node Distributed Inference MFU for Agentic AI
DataFunSummit
DataFunSummit
Jun 17, 2026 · Artificial Intelligence

Why Agentic AI Inference Is Slow and How NVIDIA Dynamo 1.1 Solves It

Developers deploying Agentic AI face multi‑turn latency caused by repeated token recomputation, KV‑cache eviction, and cold‑starts, and NVIDIA Dynamo 1.1 addresses these issues with KV‑cache‑aware routing, multi‑level cache offload, priority scheduling, and Prefill/Decode separation, as demonstrated in an upcoming Kubernetes‑based live session.

AI inferenceDistributed InferenceKV cache
0 likes · 3 min read
Why Agentic AI Inference Is Slow and How NVIDIA Dynamo 1.1 Solves It
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Jun 10, 2026 · Artificial Intelligence

LU‑KV Sets New SOTA at ICML 2026 by Redefining KV Cache Eviction

A joint effort by Baidu Baige and Fudan University introduces the LU‑KV framework, which treats KV‑cache budget allocation as a global combinatorial optimization problem, achieving only 0.52% relative performance loss at 80% compression and establishing a new efficiency‑accuracy SOTA on LongBench.

Cache EvictionICML 2026KV cache
0 likes · 5 min read
LU‑KV Sets New SOTA at ICML 2026 by Redefining KV Cache Eviction
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 7, 2026 · Artificial Intelligence

Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained

The article explains why LLM inference is memory‑bound, introduces vLLM’s three core optimizations—Continuous Batching, PagedAttention, and Prefix Caching—shows how to launch a vLLM server, run Python code to benchmark performance, and examines KV‑Cache memory usage with concrete numbers.

Continuous BatchingKV cachePagedAttention
0 likes · 11 min read
Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained
Network Intelligence Research Center (NIRC)
Network Intelligence Research Center (NIRC)
Jun 4, 2026 · Artificial Intelligence

How DeepSeek‑V4 Achieves Million‑Token Context via Aggressive KV‑Cache Compression

DeepSeek‑V4 reaches a million‑token context window by aggressively compressing its KV‑cache and employing a hybrid attention scheme that combines Compressed Sparse Attention (CSA) for selective top‑k retrieval with Heavily Compressed Attention (HCA) for full‑attention over heavily merged entries, alongside mixed‑precision storage and other engineering optimizations.

Compressed Sparse AttentionDeepSeek V4Heavily Compressed Attention
0 likes · 7 min read
How DeepSeek‑V4 Achieves Million‑Token Context via Aggressive KV‑Cache Compression
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jun 2, 2026 · Artificial Intelligence

OSCAR Beats TurboQuant: 2‑Bit KV‑Cache for Fast, Stable Long‑Context Inference

OSCAR presents an attention‑aware rotation scheme that compresses KV caches to true 2‑bit, cutting memory usage by up to 8× and boosting decode throughput by up to 7×, while preserving inference quality within a few points of BF16 across multiple models and long‑context benchmarks, outperforming TurboQuant.

2-bit quantizationKV cacheOSCAR
0 likes · 13 min read
OSCAR Beats TurboQuant: 2‑Bit KV‑Cache for Fast, Stable Long‑Context Inference
Machine Heart
Machine Heart
Jun 2, 2026 · Artificial Intelligence

Training Transformers to Be Compression‑Friendly: A New Memory‑Discard Paradigm

The article analyzes the KV‑Cache memory bottleneck of long‑context Transformers, introduces the KV‑CAT (KV‑Compression Aware Training) approach that simulates cache compression during pre‑training, and presents experiments showing unchanged base abilities while dramatically improving post‑training compression, retrieval and long‑text QA performance.

KV cacheKV-CATModel Compression
0 likes · 10 min read
Training Transformers to Be Compression‑Friendly: A New Memory‑Discard Paradigm
Old Zhang's AI Learning
Old Zhang's AI Learning
May 31, 2026 · Artificial Intelligence

vLLM 0.22 Release: Production-Ready DeepSeek V4 and Extreme KV Cache Compression

The vLLM 0.22 stable release introduces production‑grade DeepSeek V4 support, massive kernel fusions, up to 10‑20× speedups, Batch Invariance with 28.9% latency gain, a Rust front‑end, multi‑level KV cache offload that can double context length, and broad hardware coverage across NVIDIA, AMD, CPU and RISC‑V, making it a pivotal upgrade for inference infrastructure teams.

Batch InvarianceDeepSeek V4KV cache
0 likes · 13 min read
vLLM 0.22 Release: Production-Ready DeepSeek V4 and Extreme KV Cache Compression
Machine Heart
Machine Heart
May 30, 2026 · Artificial Intelligence

Can MIT’s Attention Matching Cut LLM Memory 50× Without Accuracy Loss?

MIT researchers introduce Attention Matching, a latent‑space KV‑cache compaction technique that reduces large‑language‑model memory usage up to 50‑fold with negligible precision loss, outperforming token‑pruning, summarization, and prior compaction methods across benchmarks like QuALITY, LongHealth, and AIME‑2025.

Attention MatchingKV cacheLLM
0 likes · 13 min read
Can MIT’s Attention Matching Cut LLM Memory 50× Without Accuracy Loss?
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
May 27, 2026 · Artificial Intelligence

Optimizing Large Model Inference Architecture for the Agent Era: Engineering Practices and Challenges

The article analyzes the architectural challenges of large‑model inference in the Agent era—such as memory‑intensive MLA structures, MoE communication overhead, exploding KV‑Cache size, and tool‑call accuracy—and presents a series of engineering solutions including hierarchical KV‑Cache pooling, sequence parallelism, offloading strategies, and chip‑level adaptations to achieve higher throughput and lower token costs.

AI InfraAgentDeepSeek
0 likes · 15 min read
Optimizing Large Model Inference Architecture for the Agent Era: Engineering Practices and Challenges
DataFunTalk
DataFunTalk
May 26, 2026 · Industry Insights

Why DeepSeek’s Permanent Price Cut Aims at a $10 Trillion AI Market

DeepSeek’s 75% permanent API price reduction is analyzed as a strategic move to shrink KV‑cache memory, lower hardware dependence, trigger a demand surge, reshape the AI hardware ecosystem, and capture an estimated $10 trillion market opportunity.

AI hardwareAI infrastructureAI pricing
0 likes · 13 min read
Why DeepSeek’s Permanent Price Cut Aims at a $10 Trillion AI Market
Architect
Architect
May 25, 2026 · Artificial Intelligence

From KV Cache to Harness: How DeepSeek Is Shifting Costs to the System Layer

DeepSeek’s recent V4 release shows that as model inference becomes cheaper, the dominant expenses are moving to system‑level components such as KV cache, memory, storage, compilers, scheduling, hardware adapters, and the emerging Agent Harness layer, reshaping AI infrastructure economics.

AI infrastructureAgent HarnessCost Optimization
0 likes · 23 min read
From KV Cache to Harness: How DeepSeek Is Shifting Costs to the System Layer
Old Zhang's AI Learning
Old Zhang's AI Learning
May 23, 2026 · Artificial Intelligence

The Underrated Lifesaving Template for Qwen Local Deployment

This article analyzes the hidden pitfalls of Qwen's official Jinja chat template, explains how the community‑maintained Qwen‑Fixed‑Chat‑Templates v19 fixes rendering errors, KV‑Cache loss, token waste and agent dead‑locks, and provides step‑by‑step installation instructions for LM Studio, llama.cpp, vLLM and MLX.

Agent LoopChat TemplateKV cache
0 likes · 10 min read
The Underrated Lifesaving Template for Qwen Local Deployment
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 20, 2026 · Artificial Intelligence

How New LLM Architectures Like Gemma 4 and DeepSeek V4 Cut Long‑Context Costs

The article surveys recent open‑weight LLM releases—Gemma 4, Laguna XS.2, ZAYA1‑8B and DeepSeek V4—detailing how KV‑cache sharing, per‑layer embeddings, layer‑wise attention budgeting, compressed convolutional attention and manifold‑constrained hyper‑connections dramatically reduce memory and compute for ultra‑long contexts while preserving model quality.

Attention optimizationKV cacheLLM
0 likes · 25 min read
How New LLM Architectures Like Gemma 4 and DeepSeek V4 Cut Long‑Context Costs
Old Zhang's AI Learning
Old Zhang's AI Learning
May 16, 2026 · Artificial Intelligence

vLLM 0.21.0 Arrives: Speculative Decoding Now Supports Reasoning Models

The vLLM 0.21.0 release brings five major updates—including Transformers v4 deprecation, a C++20 build requirement, KV offload with hybrid memory, speculative decoding that respects thinking budgets, and a Blackwell token‑speed backend—while offering detailed upgrade guidance for different user groups.

C++20KV cacheSpeculative Decoding
0 likes · 12 min read
vLLM 0.21.0 Arrives: Speculative Decoding Now Supports Reasoning Models
Lao Guo's Learning Space
Lao Guo's Learning Space
May 12, 2026 · Artificial Intelligence

Demystifying the Core Technologies Behind ChatGPT, GPT‑4, and DeepSeek

This article breaks down the key algorithms that power large‑language models—Transformer, Mixture‑of‑Experts, Flash Attention, KV‑Cache, Multi‑Token Prediction, quantization, Chain‑of‑Thought and Retrieval‑Augmented Generation—explaining how each contributes to the performance of ChatGPT, GPT‑4 and DeepSeek.

Chain-of-ThoughtFlash AttentionKV cache
0 likes · 10 min read
Demystifying the Core Technologies Behind ChatGPT, GPT‑4, and DeepSeek
AI Waka
AI Waka
May 12, 2026 · Artificial Intelligence

Is 3‑Bit KV Cache the Ultimate Solution? An In‑Depth Evaluation of Google’s TurboQuant

Through ten experiments on three LLMs, this study measures TurboQuant’s 3‑bit KV‑cache compression, revealing that while quality remains strong, speed gains vary by model, memory savings depend on implementation, and attention‑entropy analysis explains why 2‑bit compression degrades performance.

Attention EntropyInference PerformanceKV cache
0 likes · 14 min read
Is 3‑Bit KV Cache the Ultimate Solution? An In‑Depth Evaluation of Google’s TurboQuant
Node.js Tech Stack
Node.js Tech Stack
May 9, 2026 · Artificial Intelligence

Redis Founder Crafts DeepSeek V4 AI Inference Engine, Node.js Star Applauds

Redis creator Salvatore Sanfilippo (antirez) released DS4, a Metal‑only C inference engine tailored for DeepSeek V4 Flash on high‑end Macs, featuring narrow model focus, 2‑bit quantization, disk‑based KV cache, benchmark speeds around 26 tokens/s, and a dual OpenAI/Anthropic compatible server.

2-bit quantizationAI inference engineDeepSeek V4
0 likes · 13 min read
Redis Founder Crafts DeepSeek V4 AI Inference Engine, Node.js Star Applauds
Lao Guo's Learning Space
Lao Guo's Learning Space
May 7, 2026 · Artificial Intelligence

Gemma 4 MTP Deep Dive: Speculative Decoding & KV‑Cache Sharing for 3× Faster Inference

The article explains why large‑language‑model inference is bottlenecked by memory‑bandwidth, then details Google’s Gemma 4 MTP technique—using a small draft model with speculative decoding and shared KV‑Cache—to parallelize token prediction, achieving up to three‑fold speed gains without any loss in output quality, and provides step‑by‑step local deployment instructions.

Gemma 4KV cacheMTP
0 likes · 11 min read
Gemma 4 MTP Deep Dive: Speculative Decoding & KV‑Cache Sharing for 3× Faster Inference
Huawei Cloud Developer Alliance
Huawei Cloud Developer Alliance
Apr 29, 2026 · Artificial Intelligence

Deploy DeepSeek‑V4 on Ascend NPU with Kthena in 3 Minutes (Prefill‑Decode Separation)

This guide walks through deploying the DeepSeek‑V4‑Flash model on Ascend NPU using Kthena’s ModelRoute, detailing the Prefill‑Decode (P/D) separation architecture, KV cache transfer via Mooncake, configuration of ModelServing and ModelRoute resources, and flexible scaling of Prefill and Decode replicas for optimal performance.

Ascend NPUDeepSeek V4KV cache
0 likes · 22 min read
Deploy DeepSeek‑V4 on Ascend NPU with Kthena in 3 Minutes (Prefill‑Decode Separation)
AI Tech Publishing
AI Tech Publishing
Apr 29, 2026 · Artificial Intelligence

Why Do AI Agents Forget and Hallucinate? A Complete Guide to KV‑Cache Memory Mechanisms

The article explains that AI agents’ forgetting and hallucinations stem from token‑level attention scores causing key‑value cache eviction before retrieval, then surveys KV‑cache basics, naive growth, streaming‑LLM windowing, SnapKV’s attention‑guided compression, token‑retention studies, Memory Sparse Attention, compares these methods, and discusses practical system pitfalls and design implications.

AI AgentsKV cacheMemory Sparse Attention
0 likes · 20 min read
Why Do AI Agents Forget and Hallucinate? A Complete Guide to KV‑Cache Memory Mechanisms
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 26, 2026 · Artificial Intelligence

Why Deploying DeepSeek‑V4 Locally with vLLM Is So Challenging

The article dissects DeepSeek‑V4’s local deployment using vLLM, explaining the steep hardware requirements, the complex heterogeneous KV‑cache architecture, and the aggressive kernel‑fusion and multi‑stream optimizations that together make high‑context inference both memory‑intensive and engineering‑heavy.

DeepSeek V4GPU memoryKV cache
0 likes · 15 min read
Why Deploying DeepSeek‑V4 Locally with vLLM Is So Challenging
Architect
Architect
Apr 25, 2026 · Artificial Intelligence

DeepSeek V4: 1M‑Token Context’s Impact on Model, Inference, Cache & Agents

The DeepSeek V4 technical report shows how a 1 million‑token context forces a redesign of attention, KV‑cache, optimizer, quantization and inference budgeting, turning long‑context capability from a costly showcase into a production‑ready feature for agents, search and Chinese professional tasks.

1M contextAgentic SearchAttention optimization
0 likes · 28 min read
DeepSeek V4: 1M‑Token Context’s Impact on Model, Inference, Cache & Agents
AI Tech Publishing
AI Tech Publishing
Apr 20, 2026 · Artificial Intelligence

How Claude Code Achieves 92% Prompt Cache Hit Rate and Cuts Costs by 81% – A Deep Dive

This article explains the mechanics of prompt‑caching for large language models, breaks down static versus dynamic context, details KV‑cache operation and its pricing, and shows how Claude Code’s 30‑minute programming session reached a 92% cache hit rate that reduced inference costs by 81%, concluding with three production‑grade design rules.

AI AgentsAnthropic APIClaude Code
0 likes · 13 min read
How Claude Code Achieves 92% Prompt Cache Hit Rate and Cuts Costs by 81% – A Deep Dive
Geek Labs
Geek Labs
Apr 20, 2026 · Artificial Intelligence

A Complete Open‑Source Guide to LLM Internals: From Tokenization to Inference Optimization

This open‑source tutorial breaks down large language model internals into 11 detailed topics—covering BPE tokenization, attention mathematics, backpropagation, transformer architecture, KV‑Cache, Paged and Flash Attention, and frontier techniques—each with numeric derivations and Python code, making it ideal for developers and interview preparation.

Flash AttentionKV cacheLLM
0 likes · 5 min read
A Complete Open‑Source Guide to LLM Internals: From Tokenization to Inference Optimization
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 11, 2026 · Artificial Intelligence

Mastering SGLang: KV Cache and RadixAttention for Faster LLM Inference

This article reviews the DeepLearning.ai short course on SGLang, explains why large‑language‑model inference is slow, details how KV Cache reduces the computation from O(n²) to O(n), introduces RadixAttention for cross‑request caching, and presents code examples and benchmark results showing up to 10× speedup in real‑world RAG scenarios.

KV cachePerformance optimizationPython
0 likes · 13 min read
Mastering SGLang: KV Cache and RadixAttention for Faster LLM Inference
AI Tech Publishing
AI Tech Publishing
Apr 5, 2026 · Artificial Intelligence

Why the First Token Is Slow: A Deep Dive into KV Cache for LLM Inference

The article explains how KV cache eliminates redundant computations in autoregressive LLM generation, detailing the attention mechanism, the O(n²) waste of recomputing K and V, the cache‑based solution, its impact on time‑to‑first‑token, and the memory‑vs‑speed trade‑off.

KV cacheLLMattention
0 likes · 7 min read
Why the First Token Is Slow: A Deep Dive into KV Cache for LLM Inference
AI Step-by-Step
AI Step-by-Step
Apr 5, 2026 · Artificial Intelligence

How Context Engineering Powers Dynamic Business Data Assembly for LLM Agents

The article explains why relying solely on handcrafted prompts leads to hallucinations in LLM agents and presents six concrete context‑engineering practices—XML isolation, hierarchical ordering, KV caching, vector reranking, async memory compression, and minimal few‑shot examples—illustrated with a full e‑commerce refund‑handling case study.

AgentContext EngineeringKV cache
0 likes · 10 min read
How Context Engineering Powers Dynamic Business Data Assembly for LLM Agents
DeepHub IMBA
DeepHub IMBA
Apr 4, 2026 · Artificial Intelligence

Building Mini-vLLM from Scratch: KV‑Cache, Dynamic Batching, and Distributed Inference

This article walks through constructing Mini-vLLM, a from‑scratch LLM inference engine that tackles the O(N²) attention cost with KV‑cache, boosts throughput via dynamic batching, adds observability with Prometheus/Grafana, supports gRPC, and scales across multiple workers, with benchmark numbers demonstrating its CPU‑only performance.

DockerDynamic BatchingKV cache
0 likes · 12 min read
Building Mini-vLLM from Scratch: KV‑Cache, Dynamic Batching, and Distributed Inference
ShiZhen AI
ShiZhen AI
Apr 2, 2026 · Artificial Intelligence

How KV Cache Works and Why Large Model Outputs Cost Five Times More Than Inputs

The article explains the KV Cache mechanism that stores previously computed key/value vectors to avoid redundant Transformer calculations, delivering roughly a 5× speedup, while also detailing why generating output tokens is far more expensive than processing input tokens due to serial generation and memory trade‑offs.

KV cacheMemory Optimizationcost analysis
0 likes · 9 min read
How KV Cache Works and Why Large Model Outputs Cost Five Times More Than Inputs
SuanNi
SuanNi
Mar 26, 2026 · Artificial Intelligence

TurboQuant: Google’s 6× KV Cache Compression With Zero Accuracy Loss

TurboQuant, a new technique from Google Research, dramatically compresses key‑value caches by up to six times without precision loss, using PolarQuant and QJL algorithms to transform vectors into polar coordinates and apply quantized Johnson‑Lindenstrauss transforms, thereby boosting inference speed and enabling longer context handling for large language models.

AI compressionKV cacheTurboQuant
0 likes · 13 min read
TurboQuant: Google’s 6× KV Cache Compression With Zero Accuracy Loss
Smart Era Software Development
Smart Era Software Development
Mar 23, 2026 · Artificial Intelligence

From Context Engineering to Harness Engineering: Redefining Engineer Value in the AI Era

AI coding can generate code ten times faster, yet developers spend 70% of their time on non‑coding tasks such as testing, deployment and review, turning AI into a new bottleneck; the proposed solution, Harness Engineering, equips models with agents, KV‑Cache and multi‑agent workflows so engineers shift from writing code to designing AI‑friendly environments and orchestrating full‑lifecycle development.

AI codingAI-native toolsAgent
0 likes · 18 min read
From Context Engineering to Harness Engineering: Redefining Engineer Value in the AI Era
MaGe Linux Operations
MaGe Linux Operations
Mar 10, 2026 · Artificial Intelligence

Why Your LLM Service Hits CUDA OOM and How to Diagnose GPU Memory Issues

This guide explains the five common sources of GPU memory consumption in large‑model inference services, provides a step‑by‑step diagnosis workflow—from static usage and KV‑Cache analysis to concurrency and K8s scheduling—offers concrete command‑line checks, scripts, configuration examples, and actionable remediation and monitoring recommendations.

GPU memoryKV cacheLLM OOM
0 likes · 28 min read
Why Your LLM Service Hits CUDA OOM and How to Diagnose GPU Memory Issues
AI Explorer
AI Explorer
Mar 3, 2026 · Artificial Intelligence

How LMCache’s Lightning‑Fast KV Cache Slashes LLM First‑Token Latency

LMCache separates the KV cache from a vLLM instance into a shared service, dramatically cutting first‑token latency for repeated text, enabling multiple GPU instances to reuse cached vectors, improving hardware utilization, and supporting use cases such as long‑document QA, multi‑GPU load balancing, and prompt‑engineering, with a quick Docker‑based demo.

DockerKV cacheLMCache
0 likes · 6 min read
How LMCache’s Lightning‑Fast KV Cache Slashes LLM First‑Token Latency
DeepHub IMBA
DeepHub IMBA
Mar 3, 2026 · Artificial Intelligence

The Evolution of KV Cache Management: From Continuous Allocation to Unified Hybrid Memory Architecture

The article traces five eras of KV cache management for LLM inference—from its absence before Transformers to the emerging unified hybrid memory architecture—comparing vLLM, SGLang, and TensorRT‑LLM and offering a decision framework for selecting the right solution in various deployment scenarios.

KV cacheMemory ManagementPagedAttention
0 likes · 16 min read
The Evolution of KV Cache Management: From Continuous Allocation to Unified Hybrid Memory Architecture
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Feb 28, 2026 · Artificial Intelligence

How DualPath Revives Idle Network Cards to Break Long‑Context I/O Bottlenecks in DeepSeek V4

The article analyzes the KV‑Cache storage I/O bottleneck that limits agentic LLM inference, introduces the DualPath architecture with a storage‑to‑decode data path and RDMA‑based scheduling, and shows up to 1.87× offline and 1.96× online throughput gains on large‑scale GPU clusters.

DeepSeekDualPathKV cache
0 likes · 13 min read
How DualPath Revives Idle Network Cards to Break Long‑Context I/O Bottlenecks in DeepSeek V4
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Feb 27, 2026 · Artificial Intelligence

Can DeepSeek’s DualPath Break GPU Bottlenecks and Ignite an Agentic AI Surge?

DeepSeek’s new DualPath inference framework, co‑developed with leading Chinese universities, decouples compute from KV‑Cache memory access to eliminate I/O stalls in multi‑round agentic workloads, delivering up to nearly 2× higher throughput and dramatically reducing job‑completion time across several large‑scale LLMs.

AI infrastructureAgentic InferenceDeepSeek
0 likes · 13 min read
Can DeepSeek’s DualPath Break GPU Bottlenecks and Ignite an Agentic AI Surge?
Alibaba Cloud Developer
Alibaba Cloud Developer
Jan 26, 2026 · Artificial Intelligence

How We Scaled a 3.5B MoE LLM for Real‑Time Search Relevance

This article details the engineering challenges and solutions for deploying a 3.5 billion‑parameter MoE LLM in Taobao's search relevance pipeline, covering large‑batch scheduling, dynamic load balancing, intra‑batch KV‑Cache reuse, and MoE kernel tuning to meet sub‑second latency requirements.

KV cacheLLMMoE
0 likes · 15 min read
How We Scaled a 3.5B MoE LLM for Real‑Time Search Relevance
PaperAgent
PaperAgent
Jan 21, 2026 · Artificial Intelligence

Inside DeepSeek’s FlashMLA Update: What’s New in the MODEL1 Architecture

DeepSeek’s recent FlashMLA update introduces the new MODEL1, featuring a tighter KV-Cache layout, an extra two-stage cache, and a fixed 512×512 head dimension, with four code changes detailed in a public GitHub commit and illustrated by comparative diagrams.

AI architectureDeepSeekFlashMLA
0 likes · 3 min read
Inside DeepSeek’s FlashMLA Update: What’s New in the MODEL1 Architecture
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Dec 27, 2025 · Artificial Intelligence

Why Jeff Dean Champions Speculative Decoding: The Underlying Ideas

Jeff Dean highlighted speculative decoding as a lossless inference acceleration technique that can boost large language model throughput by 2–3×, and the article breaks down its core concepts—including parallel token verification, draft‑target model collaboration, rejection sampling theory, and practical optimizations such as continuous batching and tree‑based verification.

Continuous BatchingDraft-Target ModelInference Acceleration
0 likes · 8 min read
Why Jeff Dean Champions Speculative Decoding: The Underlying Ideas
Alibaba Cloud Infrastructure
Alibaba Cloud Infrastructure
Dec 22, 2025 · Artificial Intelligence

Boost LLM Inference with KV‑Cache‑Aware Routing on Alibaba Cloud ACK GIE

This article explains why KV‑Cache hit rate is critical for large‑model inference, describes vLLM's automatic prefix caching, outlines the distributed cache challenges, and provides a step‑by‑step guide to deploying Alibaba Cloud ACK Gateway with Inference Extension's precise‑mode prefix‑cache‑aware routing, backed by benchmark results.

Alibaba CloudKV cacheKubernetes
0 likes · 18 min read
Boost LLM Inference with KV‑Cache‑Aware Routing on Alibaba Cloud ACK GIE
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Dec 21, 2025 · Artificial Intelligence

Why KV Caching Is Critical for Efficient LLM Inference

The article breaks down the principles of KV caching in large language models, explaining how Q/K/V behavior differs between training and inference, the role of prompts, cache size trade‑offs, and the complexities of concurrent inference, all backed by concrete examples and references.

Concurrent InferenceKV cacheMemory Optimization
0 likes · 7 min read
Why KV Caching Is Critical for Efficient LLM Inference
ShiZhen AI
ShiZhen AI
Dec 4, 2025 · Artificial Intelligence

What Is a Context Window? Explaining LLM Memory Capacity

The article explains that a context window defines an LLM's token‑level memory capacity, shows how longer windows cause quadratic computation growth, introduces KV Cache as a way to extend context without exploding resources, and covers advanced techniques like Ring Attention, NIAH benchmarking, and attention decay in long sequences.

KV cacheLLMNIAH benchmark
0 likes · 6 min read
What Is a Context Window? Explaining LLM Memory Capacity
Software Engineering 3.0 Era
Software Engineering 3.0 Era
Nov 23, 2025 · Artificial Intelligence

Advanced AI Context Engineering: Building Operable Worlds (Part 2)

This article examines how to evolve AI prompt engineering into full‑stack context and environment engineering, detailing six practical design patterns from the Manus system, the limits of Vibe Coding, the Spec‑Driven development workflow, and concrete steps to give models a persistent, controllable world for long‑term tasks.

AIAgentContext Engineering
0 likes · 18 min read
Advanced AI Context Engineering: Building Operable Worlds (Part 2)
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Oct 20, 2025 · Artificial Intelligence

nanochat Source Code Deep Dive: Data Prep, Model Design, Training & Evaluation

This article revisits nanochat's core components, detailing the preparation of diverse training datasets, the scaling calculations for tokens and parameters, the model's MQA and KV‑cache design, the full training pipeline with gradient accumulation and mixed‑precision, cost breakdown, inference optimizations, evaluation tasks, and identified limitations with suggested improvements.

KV cacheLLMMQA
0 likes · 9 min read
nanochat Source Code Deep Dive: Data Prep, Model Design, Training & Evaluation
Data Party THU
Data Party THU
Oct 16, 2025 · Artificial Intelligence

How Tensor Product Attention Redefines Long‑Context Transformers

The article analyzes the Tensor Product Attention (TPA) method presented at NeurIPS 2025, explaining how it factorizes Q, K, V tensors to drastically reduce KV cache size and attention complexity, and demonstrates superior convergence, lower perplexity, and faster inference on long‑sequence tasks compared with existing attention variants.

Efficient AttentionKV cacheRoPE
0 likes · 11 min read
How Tensor Product Attention Redefines Long‑Context Transformers
Architects' Tech Alliance
Architects' Tech Alliance
Sep 30, 2025 · Artificial Intelligence

How KV Cache and CachedAttention Revolutionize LLM Inference Efficiency

This article explains how key‑value (KV) caching and the new CachedAttention technique dramatically reduce large‑language‑model inference costs by reusing stored attention data across dialogue turns, leveraging a three‑tier memory hierarchy of HBM, DRAM, and SSD to overcome bandwidth and capacity bottlenecks.

AI performanceCachedAttentionKV cache
0 likes · 8 min read
How KV Cache and CachedAttention Revolutionize LLM Inference Efficiency
Baobao Algorithm Notes
Baobao Algorithm Notes
Sep 28, 2025 · Artificial Intelligence

How Much GPU Memory Do LLMs Really Need? A Deep Dive into Training & Inference

This article breaks down the GPU memory requirements of large language models during training and inference, detailing the contributions of model weights, optimizer states, activations, KV cache, and activation recomputation, and provides concrete formulas, examples, and scaling insights for models like Qwen3 and DeepSeek V3.

GPU memoryKV cacheLLM
0 likes · 18 min read
How Much GPU Memory Do LLMs Really Need? A Deep Dive into Training & Inference
Data Party THU
Data Party THU
Sep 3, 2025 · Artificial Intelligence

Unlocking Large Model Secrets: Transformers, MoE, Fine‑Tuning, RAG & KV Caching

This article provides a comprehensive technical overview of today’s large‑model ecosystem, covering the Transformer architecture, Mixture‑of‑Experts extensions, five fine‑tuning methods, the evolution from traditional RAG to agentic RAG, classic agent design patterns, diverse text‑chunking strategies, and the KV‑cache optimization that accelerates inference.

Fine‑tuningKV cacheMixture of Experts
0 likes · 13 min read
Unlocking Large Model Secrets: Transformers, MoE, Fine‑Tuning, RAG & KV Caching
Wu Shixiong's Large Model Academy
Wu Shixiong's Large Model Academy
Aug 20, 2025 · Artificial Intelligence

Mastering Large‑Model Interview Questions: MHA, KV‑Cache, Scaled Dot‑Product, and Speculative Decoding

This guide walks through common large‑model interview challenges, including a hands‑on implementation of multi‑head attention with KV‑cache, the mathematical reason for scaling by sqrt(dₖ), a concise speculative decoding algorithm, and systematic debugging steps for NaN loss during training.

KV cacheLarge Model InterviewMulti‑Head Attention
0 likes · 14 min read
Mastering Large‑Model Interview Questions: MHA, KV‑Cache, Scaled Dot‑Product, and Speculative Decoding
Hailey Says
Hailey Says
Aug 3, 2025 · Artificial Intelligence

The 5W1H of Context Engineering: A Method for Agentic AI

This article defines Context Engineering (CE), contrasts it with traditional Prompt Engineering, explains its components and benefits for Agentic AI, outlines practical steps and best‑practice techniques—including KV‑cache design, masking, file‑system context, and attention manipulation—while also discussing evaluation challenges and future outlook.

Context EngineeringKV cacheLLM
0 likes · 18 min read
The 5W1H of Context Engineering: A Method for Agentic AI
DataFunTalk
DataFunTalk
Jul 19, 2025 · Artificial Intelligence

Mastering Context Engineering for AI Agents: KV-Cache, Tool Management, and Error Handling

Peak, co‑founder of Manus, shares practical lessons on building AI agents through context engineering, emphasizing KV‑cache optimization, stable prompt prefixes, controlled tool selection, file‑system memory, attention‑directed todo lists, and preserving error traces to improve robustness and scalability.

AI AgentsContext EngineeringError Handling
0 likes · 14 min read
Mastering Context Engineering for AI Agents: KV-Cache, Tool Management, and Error Handling
Tencent Cloud Developer
Tencent Cloud Developer
Jul 2, 2025 · Artificial Intelligence

Big Model Evolution: From Transformers to Enterprise Deployment

This article surveys the rapid evolution of large language models from the Transformer breakthrough to trillion‑parameter capabilities, explains key techniques such as self‑attention, MoE and KV‑Cache, explores practical aspects like temperature tuning, sales AI applications, and compares private versus cloud deployment strategies for enterprises.

KV cacheenterprise deploymentlarge language models
0 likes · 6 min read
Big Model Evolution: From Transformers to Enterprise Deployment
Baidu Geek Talk
Baidu Geek Talk
May 19, 2025 · Artificial Intelligence

How Baidu Cloud Achieved 4µs Low-Latency PD Inference with HPN Network Optimizations

To meet the demanding network requirements of large‑scale PD‑separated inference, Baidu Cloud built a 4 µs end‑to‑end low‑latency HPN cluster, optimized traffic management, adaptive routing, and custom Alltoall operators, resulting in up to 20 % throughput gains and reduced latency for both Prefill and Decode stages.

AI inferenceAlltoall optimizationDistributed Training
0 likes · 14 min read
How Baidu Cloud Achieved 4µs Low-Latency PD Inference with HPN Network Optimizations
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
May 16, 2025 · Artificial Intelligence

How Baidu Cloud Achieved 4µs End-to-End Latency for Large-Scale PD Inference

Baidu Intelligent Cloud built a 4µs end-to-end low‑latency HPN cluster, optimized traffic management and communication operators, and introduced dynamic expert balancing to dramatically improve the performance of large‑scale PD‑separated inference services, showcasing the deep integration of network infrastructure with AI workloads.

AI inferenceAll-to-AllHPN
0 likes · 14 min read
How Baidu Cloud Achieved 4µs End-to-End Latency for Large-Scale PD Inference
AI Algorithm Path
AI Algorithm Path
May 1, 2025 · Artificial Intelligence

Uncovering the Secrets of LLM Inference Optimization

This article dissects the major bottlenecks of large‑language‑model serving—prefill vs. decode, sparsity, memory bandwidth, KV‑cache growth—and walks through concrete engineering tricks such as paged attention, radix‑tree KV caches, compressed attention, speculative decoding, FlexGen weight scheduling, FastServe queuing, plus a runnable vLLM code snippet.

FastServeFlexGenKV cache
0 likes · 18 min read
Uncovering the Secrets of LLM Inference Optimization
AIWalker
AIWalker
Apr 2, 2025 · Artificial Intelligence

EasyControl: Plug‑and‑Play DiT Control with Arbitrary Aspect Ratios and Accelerated Inference

EasyControl introduces a lightweight condition‑injection LoRA module, a position‑aware training paradigm, and causal attention with KV‑cache to enable plug‑and‑play multi‑condition control for DiT models, supporting arbitrary image resolutions while cutting inference latency by up to 30% and preserving high‑quality generation.

Conditional GenerationDiTEasyControl
0 likes · 17 min read
EasyControl: Plug‑and‑Play DiT Control with Arbitrary Aspect Ratios and Accelerated Inference
NewBeeNLP
NewBeeNLP
Nov 18, 2024 · Artificial Intelligence

How to Optimize Multi-Head Attention: From MQA to FlashAttention and Beyond

This article examines various techniques for compressing and accelerating the KV cache in transformer models—including MQA, GQA, MLA, sliding‑window and linear attention, flash attention, page and ring attention, as well as mixed‑precision training and ZeRO parallelism—providing code snippets, implementation details, and practical trade‑offs.

FlashAttentionKV cacheattention
0 likes · 17 min read
How to Optimize Multi-Head Attention: From MQA to FlashAttention and Beyond
Baobao Algorithm Notes
Baobao Algorithm Notes
Sep 28, 2024 · Artificial Intelligence

Inside Llama 3: A Complete Guide to Modern LLM Training, Architecture, and Optimization

This article provides a thorough, yet concise, overview of Llama 3’s training pipeline, data handling, model architecture, scaling laws, post‑training techniques like SFT and DPO, and inference optimizations such as KV‑Cache, GQA, PagedAttention, and FP8 quantization, highlighting practical insights and benchmark results.

DPOKV cacheLLM training
0 likes · 32 min read
Inside Llama 3: A Complete Guide to Modern LLM Training, Architecture, and Optimization