Tagged articles
KVCache
19 articles · Page 1 of 1
DeepSeek's technical report reveals their DeepInfer framework processes 155 trillion tokens daily on just 2,400 GPUs backed by a 100 PiB SSD cache cluster, achieving $38 per GPU per hour by offloading 96% of prefill compute to storage through KVCache and SWA Bounded Replay.
AI economicsDeepInferDeepSeek
0 likes · 16 min read

Baidu Baige presents a three-layer stability framework for LLM inference services: tiered traffic control with minute-level autoscaling, dynamic request bucketing with cross-bucket KVCache migration boosting decode throughput 40-50%, and multi-level KV storage with global cache awareness keeping hit rates above 90% of theoretical maximum.
KVCacheLLM inferenceMulti-level Cache
0 likes · 21 min read

Baidu engineers detail KVCache optimizations for LLM inference: a three-layer cache-aware scheduler boosting hit rates, prefetch/interrupt mechanisms, Rolling Hash offload, Radix Tree C++ rewrite, UVA copies, MLA-specific read/write cuts, plus Kunlun XPU L2 layout tuning, DMA batching, bidirectional async, huge-page pinning, NUMA affinity, PCIe relaxed ordering, Ghost List for selective write-through, and TP-group shared buffers yielding 36% TTFT reduction.
Baidu QianfanCache-Aware SchedulingKVCache
0 likes · 21 min read

Alibaba Cloud unveiled its full-stack Agentic Storage product matrix at the 2026 Yunqi Conference, spanning AI Native storage for trillion-parameter model training and inference, and Agent Native storage for million-agent concurrency, featuring CPFS, KVCacheStore, AgenticFS, and vector memory services with concrete performance gains.
AI infrastructureAgentic StorageAgenticFS
0 likes · 9 min read

At JDD 2026, JD Advertising's Zhang Zehua details how AI agents are replacing human decision-makers across advertiser, consumer, and platform layers, showcasing the Jing Xiaotong autonomous advertising agent, a reasoning model boosting ROI 8.7%, and conversational shopping agents handling 70% vague queries via TTR and RIGER architectures.
AI AgentsAdvertising TechnologyAgentic Advertising
0 likes · 16 min read

The MiMo‑V2.5 series combines Hybrid Sliding‑Window Attention, Mixture‑of‑Experts and multimodal support with a complete redesign of KVCache management, tiered caching, prefix‑tree logic and scheduling, compressing KVCache to about one‑seventh of full‑attention models and delivering up to 40% faster Prefill, 30% lower TTFT and dramatically reduced inference costs that enable a 99% API price reduction.
Hybrid SWAInference OptimizationKVCache
0 likes · 12 min read

The article analyzes why the massive KVCache bandwidth required by heterogeneous pre‑fill/ decode (PD) separation cannot be solved at the system level, proposes a Prefill‑as‑a‑Service architecture that leverages linear‑attention models to cut KVCache generation, and validates the design with a 1‑trillion‑parameter Kimi Linear deployment that achieves 54% higher throughput and 64% lower P90 TTFT across a 100 Gbps inter‑datacenter link.
Heterogeneous PDInference OptimizationKVCache
0 likes · 7 min read

This article explains how a hierarchical sparse‑attention framework redesigns KVCache storage across GPU, CPU, and remote memory, eliminates bandwidth and capacity bottlenecks, and enables efficient inference for 128K‑token and larger contexts with dramatically reduced GPU memory usage and higher throughput.
Dynamic Sparse AttentionGPU memory optimizationHierarchical Storage
0 likes · 20 min read

This article explains the design, challenges, and high‑fidelity architecture of Tair‑KVCache‑HiSim, a simulation tool that models multi‑level KV‑Cache behavior for large‑language‑model inference, predicts latency, throughput and cost under SLO constraints, and validates its predictions against real GPU deployments with sub‑5% error.
AI infrastructureCost OptimizationKVCache
0 likes · 32 min read

This article details the architecture and implementation of Tair KVCache Manager, an enterprise‑grade service that centralises KVCache metadata, decouples inference engines from storage, provides elastic scaling, multi‑tenant isolation, high availability, and performance‑optimised cache management for large‑scale LLM inference workloads.
Cache ManagementKVCacheLLM
0 likes · 28 min read

Large language model inference faces memory pressure, but by externalizing KVCache with Mooncake and orchestrating roles via the Kubernetes‑native RoleBasedGroup (RBG), developers can achieve stable, high‑throughput, cost‑effective serving with seamless in‑place upgrades and topology‑aware performance.
AI infrastructureKVCacheKubernetes
0 likes · 21 min read

This article explains how SGLang’s hybrid model design combines Transformer attention with Mamba state‑space layers, introduces a dual‑pool memory architecture and elastic allocation, and presents specialized prefix‑cache and speculative‑decoding techniques that together enable efficient, scalable inference for long‑context large language models.
Hybrid ModelsInference OptimizationKVCache
0 likes · 22 min read

This article details the design and engineering of the 3FS distributed file system as a scalable KVCache backend for large‑language‑model inference, covering its architecture, performance tuning, reliability fixes, integration with SGLang/vLLM, and cloud‑native Kubernetes operator deployment.
3FSAI inferenceKVCache
0 likes · 30 min read

This article examines how Volcano Engine's Elastic Instant Cache (EIC) tackles the memory bottleneck, high‑concurrency latency, and cross‑node coordination challenges of large language model inference by decoupling storage and computation, pooling resources, and applying layered optimizations, ultimately boosting AI inference efficiency, scalability, and cost‑effectiveness across various deployment scenarios.
AI infrastructureDistributed CachingKVCache
0 likes · 30 min read

Mooncake, an open‑source large‑model inference platform, introduces a KVCache‑centric architecture that dramatically improves throughput, reduces latency and cuts inference costs by up to 20%, while integrating with frameworks like SGLang and vLLM and leveraging Alibaba Cloud’s eRDMA and GPUDirect technologies for scalable, high‑performance deployments.
AI performanceAlibaba CloudKVCache
0 likes · 7 min read

This article presents the YiNian LLM framework, detailing how KVCache, prefill/decoding separation, continuous batching, PageAttention, and multi‑hardware scheduling are used to speed up large language model inference while managing GPU memory and latency.
AI accelerationGPUKVCache
0 likes · 20 min read

In June 2024, Alibaba Cloud and Tsinghua University's MADSys Lab announced the open‑source Mooncake architecture, a KVCache‑centered large‑model inference framework that boosts throughput, lowers cost, and standardizes resource‑pooling techniques for high‑performance AI inference across industry and academia.
KVCacheLarge Model InferenceTsinghua University
0 likes · 4 min read

Mooncake, an open-source KVCache-centric inference architecture co-developed by Alibaba Cloud and Tsinghua University's MADSys lab, dramatically improves large-model throughput and reduces cost by decoupling resources, standardizing cache pooling, and integrating with frameworks like vLLM, sparking broad industry interest.
AI infrastructureKVCachelarge language models
0 likes · 4 min read


ArchitectJul 2, 2024 · Artificial Intelligence
The article presents Mooncake, a split‑architecture inference platform for the Kimi LLM assistant, detailing its three elastic resource pools, the rationale for using Time‑Between‑Tokens over TPOT, and design choices for Prefill, KVCache, and Decode stages to improve latency and throughput.
AI systemsDecodeKVCache
0 likes · 9 min read
