Skip to content
Tagged articles

Offloading

3 articles · Page 1 of 1
Tech Freedom Circle
Tech Freedom Circle
Nov 4, 2026 · Artificial Intelligence

Production-Grade KV Cache Optimization: Memory Budgeting, PagedAttention, Quantization & Offloading

This article provides a comprehensive guide to production-grade KV Cache optimization for LLM inference, covering memory budget formulas, PagedAttention paging mechanism, prefix caching, FP8/NVFP4 quantization, hierarchical offloading, and eviction strategies with concrete deployment priorities.

KV CacheLLM inferenceOffloading
0 likes · 32 min read
Production-Grade KV Cache Optimization: Memory Budgeting, PagedAttention, Quantization & Offloading
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Oct 28, 2026 · Artificial Intelligence

BaGuaLu-2: Runtime-Adaptive Memory Management for MoE Training on 10K+ GPU Clusters

Tsinghua University's BaGuaLu-2 system dynamically selects per-operator recomputation or offloading after MoE gating to keep activation memory within device limits, achieving 1.14× throughput over baselines on 12,288 GPUs with 32.9% average MFU, zero config changes, and open-source code.

BaGuaLuGPU clustersMoE training
0 likes · 11 min read
BaGuaLu-2: Runtime-Adaptive Memory Management for MoE Training on 10K+ GPU Clusters
Infra Learning Club
Infra Learning Club
Nov 1, 2024 · Artificial Intelligence

Configuring vLLM swap_space and cpu_offload_gb for Stable Large-Model Inference

The article explains vLLM’s GPU compute capability requirement, describes the swap_space and cpu_offload_gb parameters, outlines their ideal usage scenarios, and provides step‑by‑step code examples that demonstrate how adjusting these settings enables loading and running a 7B‑parameter model on a 16 GB T4 GPU.

GPU memory managementOffloadingcpu_offload_gb
0 likes · 9 min read
Configuring vLLM swap_space and cpu_offload_gb for Stable Large-Model Inference