BestHub
Sign in
Tagged articles

KV cache

118 articles · Page 2 of 2
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Oct 20, 2025 · Artificial Intelligence

nanochat Source Code Deep Dive: Data Prep, Model Design, Training & Evaluation

This article revisits nanochat's core components, detailing the preparation of diverse training datasets, the scaling calculations for tokens and parameters, the model's MQA and KV‑cache design, the full training pipeline with gradient accumulation and mixed‑precision, cost breakdown, inference optimizations, evaluation tasks, and identified limitations with suggested improvements.

KV cacheLLMMQA
0 likes · 9 min read
nanochat Source Code Deep Dive: Data Prep, Model Design, Training & Evaluation
Data Party THU
Data Party THU
Oct 16, 2025 · Artificial Intelligence

How Tensor Product Attention Redefines Long‑Context Transformers

The article analyzes the Tensor Product Attention (TPA) method presented at NeurIPS 2025, explaining how it factorizes Q, K, V tensors to drastically reduce KV cache size and attention complexity, and demonstrates superior convergence, lower perplexity, and faster inference on long‑sequence tasks compared with existing attention variants.

KV cacheLong ContextRoPE
0 likes · 11 min read
How Tensor Product Attention Redefines Long‑Context Transformers
Architects' Tech Alliance
Architects' Tech Alliance
Sep 30, 2025 · Artificial Intelligence

How KV Cache and CachedAttention Revolutionize LLM Inference Efficiency

This article explains how key‑value (KV) caching and the new CachedAttention technique dramatically reduce large‑language‑model inference costs by reusing stored attention data across dialogue turns, leveraging a three‑tier memory hierarchy of HBM, DRAM, and SSD to overcome bandwidth and capacity bottlenecks.

AI performanceCachedAttentionKV cache
0 likes · 8 min read
How KV Cache and CachedAttention Revolutionize LLM Inference Efficiency
Baobao Algorithm Notes
Baobao Algorithm Notes
Sep 28, 2025 · Artificial Intelligence

How Much GPU Memory Do LLMs Really Need? A Deep Dive into Training & Inference

This article breaks down the GPU memory requirements of large language models during training and inference, detailing the contributions of model weights, optimizer states, activations, KV cache, and activation recomputation, and provides concrete formulas, examples, and scaling insights for models like Qwen3 and DeepSeek V3.

GPU memoryKV cacheLLM
0 likes · 18 min read
How Much GPU Memory Do LLMs Really Need? A Deep Dive into Training & Inference
Data Party THU
Data Party THU
Sep 3, 2025 · Artificial Intelligence

Unlocking Large Model Secrets: Transformers, MoE, Fine‑Tuning, RAG & KV Caching

This article provides a comprehensive technical overview of today’s large‑model ecosystem, covering the Transformer architecture, Mixture‑of‑Experts extensions, five fine‑tuning methods, the evolution from traditional RAG to agentic RAG, classic agent design patterns, diverse text‑chunking strategies, and the KV‑cache optimization that accelerates inference.

Agentic AIFine‑tuningKV cache
0 likes · 13 min read
Unlocking Large Model Secrets: Transformers, MoE, Fine‑Tuning, RAG & KV Caching
Wu Shixiong's Large Model Academy
Wu Shixiong's Large Model Academy
Aug 20, 2025 · Artificial Intelligence

Mastering Large‑Model Interview Questions: MHA, KV‑Cache, Scaled Dot‑Product, and Speculative Decoding

This guide walks through common large‑model interview challenges, including a hands‑on implementation of multi‑head attention with KV‑cache, the mathematical reason for scaling by sqrt(dₖ), a concise speculative decoding algorithm, and systematic debugging steps for NaN loss during training.

KV cacheLarge Model InterviewMulti‑Head Attention
0 likes · 14 min read
Mastering Large‑Model Interview Questions: MHA, KV‑Cache, Scaled Dot‑Product, and Speculative Decoding
Code Archaeology
Code Archaeology
Aug 5, 2025 · Artificial Intelligence

LLM Inference Explained: Prefill, Decode, and KV Cache Mechanics

This article explains the LLM inference process, detailing the prefill and decode stages, tokenization, KV cache mechanics, and key metrics like TTFT and ITL, while introducing inference frameworks MindIE and LMDeploy with their optimization features.

DecodeKV cacheLLM inference
0 likes · 8 min read
LLM Inference Explained: Prefill, Decode, and KV Cache Mechanics
Hailey Says
Hailey Says
Aug 3, 2025 · Artificial Intelligence

The 5W1H of Context Engineering: A Method for Agentic AI

This article defines Context Engineering (CE), contrasts it with traditional Prompt Engineering, explains its components and benefits for Agentic AI, outlines practical steps and best‑practice techniques—including KV‑cache design, masking, file‑system context, and attention manipulation—while also discussing evaluation challenges and future outlook.

Agentic AIContext EngineeringKV cache
0 likes · 18 min read
The 5W1H of Context Engineering: A Method for Agentic AI
DataFunTalk
DataFunTalk
Jul 19, 2025 · Artificial Intelligence

Mastering Context Engineering for AI Agents: KV-Cache, Tool Management, and Error Handling

Peak, co‑founder of Manus, shares practical lessons on building AI agents through context engineering, emphasizing KV‑cache optimization, stable prompt prefixes, controlled tool selection, file‑system memory, attention‑directed todo lists, and preserving error traces to improve robustness and scalability.

AI AgentsContext EngineeringError Handling
0 likes · 14 min read
Mastering Context Engineering for AI Agents: KV-Cache, Tool Management, and Error Handling
Tencent Cloud Developer
Tencent Cloud Developer
Jul 2, 2025 · Artificial Intelligence

Big Model Evolution: From Transformers to Enterprise Deployment

This article surveys the rapid evolution of large language models from the Transformer breakthrough to trillion‑parameter capabilities, explains key techniques such as self‑attention, MoE and KV‑Cache, explores practical aspects like temperature tuning, sales AI applications, and compares private versus cloud deployment strategies for enterprises.

KV cacheLarge Language Modelsenterprise deployment
0 likes · 6 min read
Big Model Evolution: From Transformers to Enterprise Deployment
Baidu Geek Talk
Baidu Geek Talk
May 19, 2025 · Artificial Intelligence

How Baidu Cloud Achieved 4µs Low-Latency PD Inference with HPN Network Optimizations

To meet the demanding network requirements of large‑scale PD‑separated inference, Baidu Cloud built a 4 µs end‑to‑end low‑latency HPN cluster, optimized traffic management, adaptive routing, and custom Alltoall operators, resulting in up to 20 % throughput gains and reduced latency for both Prefill and Decode stages.

AI inferenceAlltoall optimizationHPN
0 likes · 14 min read
How Baidu Cloud Achieved 4µs Low-Latency PD Inference with HPN Network Optimizations
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
May 16, 2025 · Artificial Intelligence

How Baidu Cloud Achieved 4µs End-to-End Latency for Large-Scale PD Inference

Baidu Intelligent Cloud built a 4µs end-to-end low‑latency HPN cluster, optimized traffic management and communication operators, and introduced dynamic expert balancing to dramatically improve the performance of large‑scale PD‑separated inference services, showcasing the deep integration of network infrastructure with AI workloads.

AI inferenceAll-to-AllHPN
0 likes · 14 min read
How Baidu Cloud Achieved 4µs End-to-End Latency for Large-Scale PD Inference
AI Algorithm Path
AI Algorithm Path
May 1, 2025 · Artificial Intelligence

Uncovering the Secrets of LLM Inference Optimization

This article dissects the major bottlenecks of large‑language‑model serving—prefill vs. decode, sparsity, memory bandwidth, KV‑cache growth—and walks through concrete engineering tricks such as paged attention, radix‑tree KV caches, compressed attention, speculative decoding, FlexGen weight scheduling, FastServe queuing, plus a runnable vLLM code snippet.

FastServeFlexGenKV cache
0 likes · 18 min read
Uncovering the Secrets of LLM Inference Optimization
AIWalker
AIWalker
Apr 2, 2025 · Artificial Intelligence

EasyControl: Plug‑and‑Play DiT Control with Arbitrary Aspect Ratios and Accelerated Inference

EasyControl introduces a lightweight condition‑injection LoRA module, a position‑aware training paradigm, and causal attention with KV‑cache to enable plug‑and‑play multi‑condition control for DiT models, supporting arbitrary image resolutions while cutting inference latency by up to 30% and preserving high‑quality generation.

Conditional GenerationDiTEasyControl
0 likes · 17 min read
EasyControl: Plug‑and‑Play DiT Control with Arbitrary Aspect Ratios and Accelerated Inference
NewBeeNLP
NewBeeNLP
Nov 18, 2024 · Artificial Intelligence

How to Optimize Multi-Head Attention: From MQA to FlashAttention and Beyond

This article examines various techniques for compressing and accelerating the KV cache in transformer models—including MQA, GQA, MLA, sliding‑window and linear attention, flash attention, page and ring attention, as well as mixed‑precision training and ZeRO parallelism—providing code snippets, implementation details, and practical trade‑offs.

AttentionFlashAttentionKV cache
0 likes · 17 min read
How to Optimize Multi-Head Attention: From MQA to FlashAttention and Beyond
Baobao Algorithm Notes
Baobao Algorithm Notes
Sep 28, 2024 · Artificial Intelligence

Inside Llama 3: A Complete Guide to Modern LLM Training, Architecture, and Optimization

This article provides a thorough, yet concise, overview of Llama 3’s training pipeline, data handling, model architecture, scaling laws, post‑training techniques like SFT and DPO, and inference optimizations such as KV‑Cache, GQA, PagedAttention, and FP8 quantization, highlighting practical insights and benchmark results.

DPOKV cacheLLM training
0 likes · 32 min read
Inside Llama 3: A Complete Guide to Modern LLM Training, Architecture, and Optimization