Skip to content
Tagged articles

LLM inference

152 articles · Page 1 of 2
Machine Heart
Machine Heart
Nov 17, 2026 · Artificial Intelligence

Cache Engineering: Agents Directly Manage KV Cache to Cut Inference Costs 27.7%

Researchers from Tsinghua and CUHK propose Cache Engineering, a paradigm where agents directly manipulate KV cache via Append, Drop, and Repos operations, avoiding re-prefill overhead; their open-source AKV system reduces average inference cost by 27.7% and improves task success rate by 1.7% across four agent benchmarks.

AKVAgentCache Engineering
0 likes · 13 min read
Cache Engineering: Agents Directly Manage KV Cache to Cut Inference Costs 27.7%
Machine Heart
Machine Heart
Nov 16, 2026 · Artificial Intelligence

Latent Sparse Attention: 22x Speedup Over DeepSeek-Style at 10M Tokens

Prefrontal AI and ETH Zurich introduce Latent Sparse Attention (LSA), which clusters keys in a learned latent space and uses hierarchical bucket search to reduce indexer complexity from quadratic to O(T log T), achieving 22.7x prefill speedup over GLM-5.3-Flash and 72.4x over Qwen3.8-27B at 10M tokens while maintaining accuracy.

DeepSeekGLM-5.3-FlashLLM inference
0 likes · 18 min read
Latent Sparse Attention: 22x Speedup Over DeepSeek-Style at 10M Tokens
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 15, 2026 · Artificial Intelligence

4× DGX Spark vs M5 Ultra: Why Higher Bandwidth Mac Loses on GLM-5.3-Flash

A developer's benchmark shows four NVIDIA DGX Spark units interconnected via 200Gb/s networking outperform a single 512GB Mac Studio M5 Ultra on GLM-5.3-Flash inference across prefill, code generation, and concurrent workloads, despite the Mac's higher 1.23TB/s memory bandwidth, due to software stack differences, KV cache compression, and batching efficiency.

GLM-5.3-FlashKV CacheLLM inference
0 likes · 13 min read
4× DGX Spark vs M5 Ultra: Why Higher Bandwidth Mac Loses on GLM-5.3-Flash
Code Farming
Code Farming
Nov 15, 2026 · Artificial Intelligence

Five Cuts to Attention Optimization: MHA, MQA, GQA, FlashAttention Layer-by-Layer

This article dissects MHA, MQA, GQA, and FlashAttention across five analytical layers, revealing how structural optimizations (MQA/GQA) reduce KV Cache memory by sharing keys/values, while FlashAttention tackles memory bandwidth via tiled computation and online softmax, and why modern LLMs combine GQA with FlashAttention for 2-4x speedups without accuracy loss.

Attention MechanismFlashAttentionGQA
0 likes · 11 min read
Five Cuts to Attention Optimization: MHA, MQA, GQA, FlashAttention Layer-by-Layer
AI Frontier Lectures
AI Frontier Lectures
Nov 14, 2026 · Backend Development

llama.cpp 1.0 Drops 35 Deprecated APIs, Adopts Semantic Versioning

llama.cpp 1.0.0 removes 35 deprecated functions from its public C API, adopts semantic versioning where only breaking API changes trigger major versions, adds FP8 KV cache support, MoE CPU offloading, new model integrations including Kimi-K3.1-Pro and drama-3, and provides pre-built binaries for 150+ architecture variants across major platforms.

C APIFP8KV Cache
0 likes · 5 min read
llama.cpp 1.0 Drops 35 Deprecated APIs, Adopts Semantic Versioning
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 12, 2026 · Artificial Intelligence

ANE on Mac LLMs: 1.14x–1.49x Only for Prefill, M6 Dual 16-Core Changes Game

The article analyzes three ways to use Apple's Neural Engine (ANE) for LLM inference on Macs, revealing that 1.14x–1.49x speedups only apply to first-token latency (prefill), not token generation; M5 chips trade speed for power efficiency, while M6's dual 16-core ANE enables both speed and efficiency gains but is limited by 32GB RAM.

Apple Neural EngineCore AILLM inference
0 likes · 25 min read
ANE on Mac LLMs: 1.14x–1.49x Only for Prefill, M6 Dual 16-Core Changes Game
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 12, 2026 · Industry Insights

M5 Ultra 512GB Arrives: 768GB Vaporware, 384GB GPU Limit, Which Memory for Your LLM?

Apple's M5 Ultra Mac Studio now offers a 512GB unified memory option priced from $22,299 with 10-12 week delivery, but the 768GB tier remains unconfirmed vaporware; macOS defaults GPU wired memory to 384GB, so extra capacity only helps run larger models, not speed, and buyers should match memory to specific model sizes.

Apple SiliconGPU wired limitLLM inference
0 likes · 16 min read
M5 Ultra 512GB Arrives: 768GB Vaporware, 384GB GPU Limit, Which Memory for Your LLM?
AI Frontier Lectures
AI Frontier Lectures
Nov 10, 2026 · Artificial Intelligence

Grammar-Constrained Decoding 75x Faster: Vocabulary Compression with Equivalence Classes

Saarland University researchers present a vocabulary compression technique for grammar-constrained decoding that groups tokens into equivalence classes, reducing per-step grammar engine overhead from scanning the full vocabulary to updating only class representatives, achieving up to 75x speedup on complex grammars like Bython while preserving byte-identical output to uncompressed XGrammar2.

BythonEMNLP 2026LLM inference
0 likes · 5 min read
Grammar-Constrained Decoding 75x Faster: Vocabulary Compression with Equivalence Classes
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Nov 9, 2026 · Artificial Intelligence

MapFlow: Cross-Model Attention Map Reuse Cuts LLM Prefill Latency 15-20%

Peking University researchers introduce MapFlow, a system that reuses attention maps across different-sized LLMs to avoid recomputing shared context during model handoffs, reducing prefill latency by 15–20% and increasing multi-turn dialogue throughput up to 25% while maintaining generation quality.

LLM inferenceMapFlowPKU-DAIR
0 likes · 11 min read
MapFlow: Cross-Model Attention Map Reuse Cuts LLM Prefill Latency 15-20%
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 9, 2026 · Industry Insights

M7 Base 240GB/s vs M5 Pro 307GB/s: Apple's AI Bandwidth Claim Debunked

Analysis of Apple Silicon rumors shows M7 base model's rumored 240GB/s memory bandwidth is lower than shipping M5 Pro's 307GB/s, while M7 Ultra won't arrive until 2028 — an 18-month gap with no verified specs — making current M5 Max/Ultra the only proven high-bandwidth options for large LLM inference today.

Apple SiliconLLM inferenceM5 Pro
0 likes · 11 min read
M7 Base 240GB/s vs M5 Pro 307GB/s: Apple's AI Bandwidth Claim Debunked
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 6, 2026 · Artificial Intelligence

M6 Launch Day: 9.1 vs 10.6 tok/s — Should You Upgrade 16GB to 32GB for Local LLMs?

Real-world tests show M6 MacBook Pro hits 9.1 tok/s on Qwen3.8-27B 4-bit, reaching 86% of the 10.6 tok/s bandwidth ceiling; the article explains why the dual neural engine doesn't accelerate token generation, debunks the 8-model limit as a software filter, and provides a three-tier buying guide based on memory bandwidth and model size.

Apple SiliconBuying GuideLLM inference
0 likes · 13 min read
M6 Launch Day: 9.1 vs 10.6 tok/s — Should You Upgrade 16GB to 32GB for Local LLMs?
Ops Community
Ops Community
Nov 6, 2026 · Artificial Intelligence

Production LLM Inference with vLLM/TGI: Deployment, Tuning & Observability Guide

This comprehensive guide covers production deployment and performance tuning of LLM inference services using vLLM and TGI, including hardware planning, Docker/Kubernetes orchestration, key parameter optimization, monitoring with Prometheus, autoscaling with KEDA, troubleshooting common failures, and security hardening.

CUDA GraphDockerFP8 quantization
0 likes · 99 min read
Production LLM Inference with vLLM/TGI: Deployment, Tuning & Observability Guide
Tech Freedom Circle
Tech Freedom Circle
Nov 4, 2026 · Artificial Intelligence

Production-Grade KV Cache Optimization: Memory Budgeting, PagedAttention, Quantization & Offloading

This article provides a comprehensive guide to production-grade KV Cache optimization for LLM inference, covering memory budget formulas, PagedAttention paging mechanism, prefix caching, FP8/NVFP4 quantization, hierarchical offloading, and eviction strategies with concrete deployment priorities.

KV CacheLLM inferenceOffloading
0 likes · 32 min read
Production-Grade KV Cache Optimization: Memory Budgeting, PagedAttention, Quantization & Offloading
AI Frontier Lectures
AI Frontier Lectures
Nov 4, 2026 · Artificial Intelligence

SpecPilot: 10.87x LLM Inference Speedup via Cost-Model Guided Speculative Decoding

NVIDIA, UCB, Princeton, and Stanford introduce SpecPilot, a cost-model guided speculative decoding system that achieves 10.87x speedup over autoregressive decoding for Qwen3.8-27B on RTX PRO 6000, using 12-bit weight format, tree verification, and two-level verification to optimize per-token verification and batch scheduling.

12-bit weightsLLM inferenceNVIDIA
0 likes · 5 min read
SpecPilot: 10.87x LLM Inference Speedup via Cost-Model Guided Speculative Decoding
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 1, 2026 · Artificial Intelligence

Two RTX 3090s Hit 3145 tok/s on 180B Model: Strata Engine Brings Giant LLMs to Gaming PCs

The Strata inference engine enables Qwen3.8-Flash-Next (180B total params, 6B active) to achieve 3145 tokens/sec prefill and 120 tokens/sec decode at 400K context on dual RTX 3090s by offloading experts to CPU/SSD, using speculative decoding and chunked prefill, offering a low-cost alternative to Mac Studio for long-context workloads.

LLM inferenceQwen3.8-Flash-NextRTX 3090
0 likes · 11 min read
Two RTX 3090s Hit 3145 tok/s on 180B Model: Strata Engine Brings Giant LLMs to Gaming PCs
Lao Guo's Learning Space
Lao Guo's Learning Space
Oct 31, 2026 · Artificial Intelligence

552B DeepSeek on M5 Ultra 256GB: 3883 Prefill, 92 Decode Tok/s – Local Full-Scale LLM Achieved

Community developers benchmarked the 552B DeepSeek V4.1-Flash model at 3bpw quantization on a Mac Studio M5 Ultra with 256GB unified memory, achieving 3883 tok/s prefill, 92 tok/s decode, and up to 163 tok/s with speculative decoding, demonstrating that full-scale LLMs can run locally on consumer hardware.

Apple SiliconDeepSeekLLM inference
0 likes · 8 min read
552B DeepSeek on M5 Ultra 256GB: 3883 Prefill, 92 Decode Tok/s – Local Full-Scale LLM Achieved
Ops Community
Ops Community
Oct 28, 2026 · Backend Development

vLLM Production Tuning: The max_num_seqs, GPU Memory & Concurrency Triad

This comprehensive guide explains how vLLM's three core parameters — gpu_memory_utilization, max_model_len, and max_num_seqs — form an interdependent constraint triangle, and provides formulas, profiling log analysis, benchmarking methods, deployment configs, and five real-world troubleshooting cases to systematically tune LLM inference services for latency, throughput, and stability.

GPU memory managementKV CacheKubernetes Deployment
0 likes · 67 min read
vLLM Production Tuning: The max_num_seqs, GPU Memory & Concurrency Triad
Lao Guo's Learning Space
Lao Guo's Learning Space
Oct 28, 2026 · Artificial Intelligence

M6 Mac Memory Tiers: Max LLM Size & Speed for 16/24/32GB (Real Tests)

This article analyzes Apple M6 Mac memory configurations (16GB/153GB/s, 24GB/170GB/s, 32GB/170GB/s) for local LLM inference, using official specs and community benchmarks to determine which models (7B–32B) fit each tier, their token speeds, and buying recommendations for different user needs.

Apple M6Apple SiliconBuying Guide
0 likes · 8 min read
M6 Mac Memory Tiers: Max LLM Size & Speed for 16/24/32GB (Real Tests)
Lao Guo's Learning Space
Lao Guo's Learning Space
Oct 26, 2026 · Artificial Intelligence

M5 Max 128GB: Running 125B Qwen3.8-Flash-Next at 60 tok/s with MLX & MTP

This guide details running Qwen3.8-Flash-Next 125B MoE on Apple M5 Max 128GB, covering hardware bandwidth limits, memory constraints, framework selection (MLX vs llama.cpp), command-level optimizations including MTP speculative decoding and GPU memory limit tuning, and realistic performance expectations of 45–60 tok/s with MTP enabled.

Apple SiliconLLM inferenceM5 Max
0 likes · 9 min read
M5 Max 128GB: Running 125B Qwen3.8-Flash-Next at 60 tok/s with MLX & MTP
DeepHub IMBA
DeepHub IMBA
Oct 26, 2026 · Backend Development

vLLM Adds FP8 W8A8 Kernel for AMD Strix Halo, Up to 2.54x GEMM Speedup

vLLM's PR #63199 introduces a dedicated FP8 W8A8 Blockscale GEMM kernel for AMD's gfx1151 (RDNA 3.5) architecture, accelerating specific small-batch matrix multiplications by 1.70–2.54x and improving end-to-end generation throughput by ~6.8% on Ryzen AI MAX+ 395, though the PR remains in draft and covers only a subset of FP8 operations.

AMDFP8GEMM
0 likes · 8 min read
vLLM Adds FP8 W8A8 Kernel for AMD Strix Halo, Up to 2.54x GEMM Speedup
Old Zhang's AI Learning
Old Zhang's AI Learning
Oct 25, 2026 · Artificial Intelligence

vLLM & SGLang Surpass DeepSeek's DeepInfer: General Engines Prevail

vLLM and SGLang matched or exceeded DeepSeek's custom DeepInfer engine on DeepSeek-V4.1-Flash within days by adopting its open-sourced operators and optimizing for model-specific traits, proving general-purpose inference engines remain dominant due to full production-grade systems.

DeepInferDeepSeekKV Cache
0 likes · 15 min read
vLLM & SGLang Surpass DeepSeek's DeepInfer: General Engines Prevail
Ops Community
Ops Community
Oct 24, 2026 · Operations

High Availability for LLM Inference: Health Checks, Warmup, Failover & Degradation

This article details a production-grade high-availability framework for vLLM-based LLM inference services on Kubernetes, covering layered health checks, multi-level model warmup, automated failover, degradation strategies, metrics-driven alerting, and troubleshooting patterns derived from real incidents.

FailoverGPUHigh Availability
0 likes · 57 min read
High Availability for LLM Inference: Health Checks, Warmup, Failover & Degradation
Lao Guo's Learning Space
Lao Guo's Learning Space
Oct 23, 2026 · Artificial Intelligence

128GB vs 256GB: Which LLMs Actually Fit on RTX Spark and Mac Studio M5 Ultra?

The article compares unified memory capacities of NVIDIA RTX Spark (128GB) and Apple Mac Studio M5 Ultra (256GB) for local LLM inference, calculating model fit at Q4 quantization (0.5 bytes/parameter), categorizing models by size, noting Spark's clustering via ConnectX-7 for scaling, and concluding capacity needs depend on target model size.

ConnectX-7LLM inferenceMac Studio M5 Ultra
0 likes · 6 min read
128GB vs 256GB: Which LLMs Actually Fit on RTX Spark and Mac Studio M5 Ultra?
DeepHub IMBA
DeepHub IMBA
Oct 22, 2026 · Artificial Intelligence

Gufo's XDNA2 NPU Offload for LLM Prefill: 5-10% Speedup, Numerical Accuracy Issues

Gufo's experimental branch enables GPU-NPU cooperative inference on AMD Ryzen AI MAX+ 395, offloading select Prefill GEMM operations to the XDNA2 NPU for 5-10% throughput gains, but introduces numerical errors reducing Top-1 token consistency to ~94% due to 8-bit block floating-point accumulation with repeated BF16 rounding.

AMD Ryzen AI MAX+ 395GPU-NPU cooperationGufo
0 likes · 9 min read
Gufo's XDNA2 NPU Offload for LLM Prefill: 5-10% Speedup, Numerical Accuracy Issues
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Oct 21, 2026 · Artificial Intelligence

DeepSeek Open-Sources DeepInfer Inference Engine & deepseek-recipe API Adapter

DeepSeek has open-sourced DeepInfer, a high-performance inference framework supporting distributed inference, speculative decoding, and SWA Bounded Replay for MoE models, alongside deepseek-recipe, a production-grade token encoding/decoding component that standardizes API request handling for consistent deployment.

API adapterDeepInferDeepSeek
0 likes · 9 min read
DeepSeek Open-Sources DeepInfer Inference Engine & deepseek-recipe API Adapter
AI Programming Lab
AI Programming Lab
Oct 21, 2026 · Artificial Intelligence

DeepSeek DeepInfer: How 2,400 GPUs Earn $38/Hour via SSD Cache Offload

DeepSeek's technical report reveals their DeepInfer framework processes 155 trillion tokens daily on just 2,400 GPUs backed by a 100 PiB SSD cache cluster, achieving $38 per GPU per hour by offloading 96% of prefill compute to storage through KVCache and SWA Bounded Replay.

AI economicsDeepInferDeepSeek
0 likes · 16 min read
DeepSeek DeepInfer: How 2,400 GPUs Earn $38/Hour via SSD Cache Offload
Old Zhang's AI Learning
Old Zhang's AI Learning
Oct 19, 2026 · Artificial Intelligence

vLLM 0.32.0 Adds System 1 Decision Engine for Millisecond Structured Decisions

vLLM 0.32.0 introduces a built-in /v1/systemone endpoint that delivers millisecond-level structured decisions (choice, boolean, scoring) using models like LiquidAI/d1-3B, achieving 23ms median latency and 44 decisions per second, alongside KV cache quantization improvements, universal HF parsers, and breaking changes.

HF tool parserKV cache quantizationLLM inference
0 likes · 11 min read
vLLM 0.32.0 Adds System 1 Decision Engine for Millisecond Structured Decisions
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Oct 18, 2026 · Artificial Intelligence

Stabilizing LLM Inference: Traffic Shaping, Dynamic Load Balancing & KVCache Awareness

Baidu Baige presents a three-layer stability framework for LLM inference services: tiered traffic control with minute-level autoscaling, dynamic request bucketing with cross-bucket KVCache migration boosting decode throughput 40-50%, and multi-level KV storage with global cache awareness keeping hit rates above 90% of theoretical maximum.

KVCacheLLM inferenceMulti-level Cache
0 likes · 21 min read
Stabilizing LLM Inference: Traffic Shaping, Dynamic Load Balancing & KVCache Awareness
Architects' Tech Alliance
Architects' Tech Alliance
Oct 18, 2026 · Artificial Intelligence

HBF High-Bandwidth Flash for LLM Inference: Simulation Reveals Benefits and Hard Technical Barriers

This article evaluates High Bandwidth Flash (HBF) as a supplement to HBM for LLM inference through simulation, showing HBF can reduce GPU count and boost throughput for long-context and MoE models, but identifies three critical barriers: read bandwidth must match HBM, flash endurance needs 25× improvement, and write latency must be lowered for practical deployment.

HBFHBMHigh Bandwidth Flash
0 likes · 6 min read
HBF High-Bandwidth Flash for LLM Inference: Simulation Reveals Benefits and Hard Technical Barriers
Wu Shixiong's Large Model Academy
Wu Shixiong's Large Model Academy
Oct 16, 2026 · Artificial Intelligence

Debugging LLM Adapter Parameter Passing: Config vs. Generate Call Mismatch

The article reveals how LLM generation parameters like repetition_penalty and no_repeat_ngram_size can be lost in adapter layers, showing through local HF and API provider case studies that configuration dictionaries often don't fully propagate to the actual generate call, and emphasizes verifying parameters at the call boundary rather than trusting logs.

API IntegrationDebuggingHugging Face
0 likes · 16 min read
Debugging LLM Adapter Parameter Passing: Config vs. Generate Call Mismatch
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Oct 15, 2026 · Artificial Intelligence

TB-Scale LLM Inference: Zero-Redundancy Weight Deployment & 80% KV Cache Compression

Baidu Baige presents extreme VRAM optimization for TB-scale model inference: zero-redundancy weight deployment via DP Attn-sharding (releasing 16–30 GiB/card) and full-route CPP (50% throughput gain), plus KV Cache compression via LU-KV (80% reduction, 0.52% accuracy loss), Draft SWA, and PD handshake optimization (20% VRAM release).

KV Cache compressionLLM inferenceLU-KV
0 likes · 27 min read
TB-Scale LLM Inference: Zero-Redundancy Weight Deployment & 80% KV Cache Compression
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Oct 14, 2026 · Artificial Intelligence

Baidu's KVCache Overhaul: 36% TTFT Cut via Hit-Rate & Multi-Chip Tuning

Baidu engineers detail KVCache optimizations for LLM inference: a three-layer cache-aware scheduler boosting hit rates, prefetch/interrupt mechanisms, Rolling Hash offload, Radix Tree C++ rewrite, UVA copies, MLA-specific read/write cuts, plus Kunlun XPU L2 layout tuning, DMA batching, bidirectional async, huge-page pinning, NUMA affinity, PCIe relaxed ordering, Ghost List for selective write-through, and TP-group shared buffers yielding 36% TTFT reduction.

Baidu QianfanCache-Aware SchedulingKVCache
0 likes · 21 min read
Baidu's KVCache Overhaul: 36% TTFT Cut via Hit-Rate & Multi-Chip Tuning
Old Zhang's AI Learning
Old Zhang's AI Learning
Oct 7, 2026 · Artificial Intelligence

AI-Generated Inference Engine Beats vLLM by 90% at 1,792 tokens/s

Baseten engineers used Claude Code to auto-generate a custom inference engine (VibeQwen) for Qwen-3.6-35B-A3B on B200, achieving 1,792 tokens/s decode (90% faster than vLLM), 12ms TTFT, and 10,307 tokens/s throughput at concurrency 32, via MetaInfer's LLM-as-Compiler loop with accuracy oracle and reusable optimizations, also speeding up SAM 3.1 by 50% in two days.

AI-generated codeB200 GPUClaude Code
0 likes · 11 min read
AI-Generated Inference Engine Beats vLLM by 90% at 1,792 tokens/s
Ops Development & AI Practice
Ops Development & AI Practice
Sep 26, 2026 · Artificial Intelligence

Mac mini M6 32GB LLM Inference: Real TPS, Bandwidth Limits & Hybrid Cloud Strategy

This analysis dissects the 32GB Mac mini M6's true LLM inference capabilities, revealing 170 GB/s unified memory bandwidth limits, GPU compute constraints versus AMD Strix Halo, measured tokens-per-second for 8B–32B models, and a cost-driven argument for hybrid local-cloud deployment over expensive local-only workstations.

Apple SiliconLLM inferenceM6 chip
0 likes · 21 min read
Mac mini M6 32GB LLM Inference: Real TPS, Bandwidth Limits & Hybrid Cloud Strategy
Ops Development & AI Practice
Ops Development & AI Practice
Sep 26, 2026 · Artificial Intelligence

Local LLM Hardware Truths: MoE, CPU/GPU/NPU, and 32GB Mac Mini Tuning

This article explains why CPUs struggle with MoE model prefill but excel at decode, compares dense, SSM, MLA, and BitNet architectures, evaluates CPU, NVIDIA GPU, Apple UMA, and NPU hardware for local LLM inference, and provides a practical tuning guide for running 32B models on a 32GB Mac mini with MLX and Ollama.

Apple SiliconKV CacheLLM inference
0 likes · 21 min read
Local LLM Hardware Truths: MoE, CPU/GPU/NPU, and 32GB Mac Mini Tuning
Lao Guo's Learning Space
Lao Guo's Learning Space
Sep 23, 2026 · Artificial Intelligence

Mac mini 16GB to Studio 256GB: Which Mac Runs Your LLMs Best?

This article analyzes Mac mini and Mac Studio configurations for local LLM inference, showing how unified memory capacity determines which models fit and memory bandwidth dictates token generation speed, with real-world benchmarks for 8B to 235B models across M6, M5 Pro, M5 Max, and M5 Ultra chips.

Apple SiliconLLM inferenceMac Studio
0 likes · 13 min read
Mac mini 16GB to Studio 256GB: Which Mac Runs Your LLMs Best?
Lao Guo's Learning Space
Lao Guo's Learning Space
Sep 23, 2026 · Artificial Intelligence

M5 Ultra 256GB vs RTX 5090 32GB: Local LLM Inference Compared

This article compares Apple M5 Ultra (256GB unified memory) and NVIDIA RTX 5090 (32GB GDDR7) for local LLM inference across 27B–100B+ models, measuring capacity limits, token throughput, power draw, and CUDA vs MLX ecosystems, concluding M5 Ultra excels beyond 32GB while RTX 5090 leads on smaller models.

CUDALLM inferenceM5 Ultra
0 likes · 10 min read
M5 Ultra 256GB vs RTX 5090 32GB: Local LLM Inference Compared
Tencent Technical Engineering
Tencent Technical Engineering
Sep 22, 2026 · Backend Development

Tencent Backend Veteran's Reset: Mastering LLM Inference Engineering from Zero

A Tencent senior backend engineer with 15 years experience shares his transition to LLM inference engineering, detailing learning paths for model architecture, inference optimization, and self-assessment exercises, while arguing that traditional infrastructure skills remain crucial for AI Infra and Agent Infra development.

AI infrastructureBackend EngineeringKV Cache
0 likes · 38 min read
Tencent Backend Veteran's Reset: Mastering LLM Inference Engineering from Zero
Lao Guo's Learning Space
Lao Guo's Learning Space
Sep 22, 2026 · Industry Insights

M5 Ultra 256GB Benchmarks: 27B LLM 51 tok/s, GPU +82% in Cyberpunk 4K RT

Hands-on benchmarks of the M5 Ultra 256GB Mac Studio show 45.7% faster single-stream LLM decoding, 3.6x higher multi-user prefill throughput, 43.6% CPU multi-core gains, 67.8% GPU improvement, and 82.8% higher Cyberpunk 2077 4K ray-tracing frame rates versus M3 Ultra, with purchasing guidance.

GPU performanceLLM inferenceM3 Ultra
0 likes · 13 min read
M5 Ultra 256GB Benchmarks: 27B LLM 51 tok/s, GPU +82% in Cyberpunk 4K RT
Lao Guo's Learning Space
Lao Guo's Learning Space
Sep 22, 2026 · Industry Insights

M5 Ultra 64 vs 80 Core GPU: What ¥9750 Buys – Not Faster LLM Decoding

The article compares Apple M5 Ultra 64-core and 80-core GPU variants, revealing that the ¥9750 price difference adds 16 GPU cores, 6 CPU cores, and 512GB memory support, but identical 1.2TB/s memory bandwidth means LLM decoding speed remains unchanged; the upgrade only benefits GPU-intensive tasks like 3D rendering and video effects.

3D renderingApple M5 UltraBuying Guide
0 likes · 10 min read
M5 Ultra 64 vs 80 Core GPU: What ¥9750 Buys – Not Faster LLM Decoding
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Sep 21, 2026 · Artificial Intelligence

Random KV Cache Eviction Rivals Top Baselines, Boosts Throughput 43%

Salesforce AI Research and UIUC propose Random Attention, a simple KV cache eviction method that randomly retains reasoning tokens while protecting the prompt, matching state-of-the-art baselines across math, science, and code reasoning tasks and improving inference throughput by 32–43% in vLLM serving.

Cache EvictionKV CacheLLM inference
0 likes · 20 min read
Random KV Cache Eviction Rivals Top Baselines, Boosts Throughput 43%
Lao Guo's Learning Space
Lao Guo's Learning Space
Sep 21, 2026 · Artificial Intelligence

M5 Ultra 256GB: Which LLMs Fit? 27B to 200B+ Capacity Analysis

The article analyzes Apple M5 Ultra 256GB unified memory capacity for local LLM inference, showing Q4-quantized models from 27B to 200B+ fit, but KV cache overhead for long contexts reduces headroom; it compares with multi-GPU setups and advises on purchase decisions.

Apple SiliconKV CacheLLM inference
0 likes · 8 min read
M5 Ultra 256GB: Which LLMs Fit? 27B to 200B+ Capacity Analysis
Machine Heart
Machine Heart
Sep 19, 2026 · Artificial Intelligence

Random Attention: Random KV Cache Eviction Rivals Best Baselines, Boosts Throughput 43%

Salesforce and UIUC researchers propose Random Attention, a simple KV cache eviction method that randomly retains reasoning tokens while fully protecting the prompt, matching the accuracy of complex importance-based methods across multiple models and tasks while increasing inference throughput by 32-43% in vLLM serving.

Attention MechanismCache CompressionEfficient AI
0 likes · 18 min read
Random Attention: Random KV Cache Eviction Rivals Best Baselines, Boosts Throughput 43%
StarRocks
StarRocks
Sep 15, 2026 · Databases

StarRocks AI Function: Embedding LLM Inference into the SQL Execution Pipeline

StarRocks introduces AI Functions that let SQL directly call LLMs for generation, classification, extraction, and embedding, integrating model inference into the query optimizer and pipeline execution engine with async dual-pipeline architecture, backpressure, and cost-based optimization to reduce unnecessary calls and manage latency.

AI FunctionLLM inferenceSQL execution engine
0 likes · 25 min read
StarRocks AI Function: Embedding LLM Inference into the SQL Execution Pipeline
Architects' Tech Alliance
Architects' Tech Alliance
Sep 5, 2026 · Artificial Intelligence

CloudMatrix384: Co-Designing Supernodes for Trillion-Parameter LLM Inference

Huawei's CloudMatrix384 integrates 384 Ascend 910 NPUs with a unified bus network and decoupled PDC service architecture to achieve 4.45 tokens/s/TFLOPS prefill efficiency on DeepSeek-R1 671B, outperforming H100 baselines through fused MoE communication, INT8 quantization, and heterogeneous pipelining.

Ascend 910CloudMatrix384Expert Parallelism
0 likes · 20 min read
CloudMatrix384: Co-Designing Supernodes for Trillion-Parameter LLM Inference
DeepHub IMBA
DeepHub IMBA
Sep 3, 2026 · Artificial Intelligence

FlashSpec: Adaptive LLM Inference with Speculative Decoding — Six Hard-Won Lessons

FlashSpec implements speculative decoding using GPU-native Triton kernel verification and online bandit-based draft model selection, sharing six practical lessons on specification-first development, hidden temperature bugs, cross-platform packaging pitfalls, kernel performance trade-offs, property-based testing value, and adaptive algorithm prerequisites.

CI/CDLLM inferenceThompson sampling
0 likes · 15 min read
FlashSpec: Adaptive LLM Inference with Speculative Decoding — Six Hard-Won Lessons
AI Engineering
AI Engineering
Aug 28, 2026 · Cloud Native

Why Round‑Robin Fails for LLM Inference and How llm‑d Fixes It

Round‑robin routing in Kubernetes wipes out KV‑cache benefits for LLM inference, but llm‑d introduces cache‑aware routing, hierarchical eviction, and prefill/decode separation, delivering up to three‑fold throughput gains and halving first‑token latency, as shown in Tesla's production rollout.

KV CacheKubernetesLLM inference
0 likes · 7 min read
Why Round‑Robin Fails for LLM Inference and How llm‑d Fixes It
ThinkingAgent
ThinkingAgent
Aug 24, 2026 · Artificial Intelligence

Why Quantization and KV‑Cache Are Key to High‑Performance LLM Inference

The article analyzes why the same LLM can exhibit vastly different cost, speed, and concurrency across inference systems, showing that KV‑cache memory management, continuous batching, PagedAttention, quantization trade‑offs, and speculative decoding together determine real‑world throughput and latency.

KV CacheLLM inferencecontinuous batching
0 likes · 39 min read
Why Quantization and KV‑Cache Are Key to High‑Performance LLM Inference
AI Architecture Path
AI Architecture Path
Aug 7, 2026 · Artificial Intelligence

Run 2.8‑Trillion‑Parameter Models on a 4 GB GPU with AirLLM’s Lossless Inference

AirLLM lets ordinary 4‑GB consumer GPUs run massive LLMs such as 70B Llama and the 2.8‑trillion‑parameter Kimi K3 MoE without quantisation, by streaming model layers from disk, offering a detailed comparison with traditional low‑VRAM tricks, step‑by‑step installation, pitfalls, scenario‑based guidance and an assessment of strengths and risks.

AirLLMLLM inferenceMoE
0 likes · 16 min read
Run 2.8‑Trillion‑Parameter Models on a 4 GB GPU with AirLLM’s Lossless Inference
DeepHub IMBA
DeepHub IMBA
Aug 3, 2026 · Artificial Intelligence

How Continuous Batching Boosts LLM Throughput by Up to 36×

The article explains continuous batching, a scheduling technique that re‑evaluates the batch at every inference iteration, eliminating the straggler problem of static batching, and shows how it, together with PagedAttention, can raise LLM service throughput dramatically while balancing latency and memory usage.

GPU utilizationLLM inferencecontinuous batching
0 likes · 9 min read
How Continuous Batching Boosts LLM Throughput by Up to 36×
AI Engineer Programming
AI Engineer Programming
Aug 2, 2026 · Artificial Intelligence

Comprehensive Cost Assessment of End-to-End RAG Systems

This report breaks down production‑grade Retrieval‑Augmented Generation (RAG) system costs into five modules—LLM inference, vector database, embedding, bandwidth, and infrastructure—revealing that model choice drives over 40% of expenses, quantisation can halve vector costs, and multimodal storage may outpace vector database spending.

EmbeddingLLM inferenceRAG
0 likes · 14 min read
Comprehensive Cost Assessment of End-to-End RAG Systems
MaGe Linux Operations
MaGe Linux Operations
Jul 21, 2026 · Cloud Native

Auto‑Scaling LLM Inference with Kubernetes HPA Based on Request Queue Depth

The article explains how to replace CPU‑only autoscaling for large‑model inference services with a Kubernetes HPA that scales pods according to a custom queue‑depth metric exported to Prometheus, covering metric definition, deployment configuration, Prometheus‑Adapter setup, HPA creation, capacity calculation, validation, troubleshooting, and rollback procedures.

HPAKubernetesLLM inference
0 likes · 21 min read
Auto‑Scaling LLM Inference with Kubernetes HPA Based on Request Queue Depth
Machine Heart
Machine Heart
Jul 20, 2026 · Artificial Intelligence

Li Xiuhong on Cross-Cluster Heterogeneous PD Separation and Token Factory “Super Pipeline” at WAIC

The article details how 无问芯穹’s Agentic Infra strategy uses a cross‑cluster heterogeneous PD‑separation architecture (PDD) to cut first‑token latency by 51.5% and token cost by 37.5%, explains the bandwidth bottleneck of KV‑Cache transfer, introduces Decode‑side RadixCache and three‑stage handoff mechanisms, and shows a 37.5% BCR improvement that translates into roughly ten‑fold inference cost reduction.

Agentic InfraCross-ClusterLLM inference
0 likes · 25 min read
Li Xiuhong on Cross-Cluster Heterogeneous PD Separation and Token Factory “Super Pipeline” at WAIC
MaGe Linux Operations
MaGe Linux Operations
Jul 19, 2026 · Operations

How to Deploy vLLM for an OpenAI‑Compatible Inference Service

This guide walks through deploying vLLM on Linux with NVIDIA GPUs and Docker Compose, covering service boundaries, host and container checks, model directory validation, image and parameter verification, minimal startup, Compose configuration, API testing, concurrency tuning, multi‑GPU troubleshooting, Nginx exposure, upgrade/rollback procedures, and daily health checks.

Docker ComposeGPUKV Cache
0 likes · 18 min read
How to Deploy vLLM for an OpenAI‑Compatible Inference Service
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
Jul 16, 2026 · Artificial Intelligence

Cut First‑Token Latency by 3.25×: Introducing HYPIC for Position‑Independent Caching in Hybrid‑Attention LLMs

HYPIC combines position‑independent caching with hybrid‑attention LLMs, reducing first‑token latency by 3.25× and sustaining QPS by 1.66× while keeping quality loss under 2 points, through a segment‑cumulative transition operator, a seam‑window fix for full‑attention layers, and segment‑parallel execution.

HYPICKV CacheLLM inference
0 likes · 13 min read
Cut First‑Token Latency by 3.25×: Introducing HYPIC for Position‑Independent Caching in Hybrid‑Attention LLMs
Machine Heart
Machine Heart
Jul 16, 2026 · Artificial Intelligence

Brake Overthinking in Long‑Reasoning Models by Detecting Semantic Redundancy

Long‑thinking LLMs often waste 41‑52% of tokens after the final answer; the PUMA framework detects when reasoning stops producing new semantic information, enabling early exit that cuts average token usage by 26.2% while keeping accuracy stable and even improving speed across multiple benchmarks.

LLM inferencePUMAToken Reduction
0 likes · 9 min read
Brake Overthinking in Long‑Reasoning Models by Detecting Semantic Redundancy
AI Engineer Programming
AI Engineer Programming
Jul 14, 2026 · Backend Development

How vLLM Generates Tokens: A Deep Dive into the Source Code

This article walks through vLLM’s token‑generation pipeline by dissecting its source files, process roles, ZMQ communication, scheduling, and deployment considerations, revealing how prompts become streamed token IDs and highlighting the key factors that limit concurrency and performance.

GPU memoryLLM inferenceZMQ
0 likes · 17 min read
How vLLM Generates Tokens: A Deep Dive into the Source Code
Alimama Tech
Alimama Tech
Jul 8, 2026 · Artificial Intelligence

Optimizing Qwen Series Inference with SGLang‑ATOM on AMD Instinct GPUs

Alibaba Mama AI Infra and AMD collaborated to integrate SGLang‑ATOM plugin mode for Qwen 3.5/3.6 models on MI308X GPUs, detailing multimodal, GDN, KV‑cache, and parallel‑strategy optimizations that achieve low‑latency, high‑throughput inference suitable for complex advertising workloads.

AMD ATOMGPU optimizationLLM inference
0 likes · 18 min read
Optimizing Qwen Series Inference with SGLang‑ATOM on AMD Instinct GPUs
Data Party THU
Data Party THU
Jul 7, 2026 · Artificial Intelligence

Parallel Decoding for Large Language Models: Balancing Inference Speed and Sampling Diversity in E‑GRM

The article presents an engineering analysis of the E‑GRM framework, detailing how parallel decoding, a temperature‑ladder sampling strategy, and batch‑parallel KV‑Cache sharing achieve low‑latency, high‑diversity inference while preserving consensus‑driven routing accuracy.

Batch ParallelismConsensus RoutingE‑GRM
0 likes · 14 min read
Parallel Decoding for Large Language Models: Balancing Inference Speed and Sampling Diversity in E‑GRM
AI Tech Publishing
AI Tech Publishing
Jun 29, 2026 · Artificial Intelligence

Understanding the Full LLM Inference Pipeline: From Tokenization to Streaming Output

The article walks through every stage of LLM inference—from tokenization and embedding, through transformer layers, prefill (compute‑bound) and decode (memory‑bound) phases, KV‑cache management, attention redesign, quantization, and serving infrastructure—explaining how each step consumes time and resources and how to identify bottlenecks.

Attention optimizationDecodeKV Cache
0 likes · 14 min read
Understanding the Full LLM Inference Pipeline: From Tokenization to Streaming Output
DataFunTalk
DataFunTalk
Jun 29, 2026 · Artificial Intelligence

DSpark Explained: 10 Key Concepts You Need to Know

The DSpark system from DeepSeek combines batch decoding, speculative decoding, draft‑model tricks, Eagle‑MTP, DFlash parallelism, variable‑length scheduling and online confidence calibration to deliver up to 85% speedup and four‑fold throughput gains while maintaining generation quality.

Batch DecodingDFlashDSpark
0 likes · 12 min read
DSpark Explained: 10 Key Concepts You Need to Know
PaperAgent
PaperAgent
Jun 29, 2026 · Artificial Intelligence

DeepSeek Opens DSpark: A New Speculative Decoding Framework for Large Language Models

DeepSeek releases DSpark, an open‑source speculative decoding system that combines semi‑autoregressive generation with confidence‑scheduled verification, delivering 60‑85% per‑user speed gains, lower latency, and superior acceptance rates compared with Eagle3 and DFlash across multiple LLM benchmarks.

Confidence SchedulingLLM inferenceSemi-Autoregressive Generation
0 likes · 14 min read
DeepSeek Opens DSpark: A New Speculative Decoding Framework for Large Language Models
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jun 28, 2026 · Artificial Intelligence

DSpark Explained in 10 Essential Concepts: System‑Level Engineering Insights

DSpark, DeepSeek’s new LLM inference framework, combines batch processing, speculative decoding, Eagle‑style draft models and DFlash‑style parallel generation with a lightweight sequential head and hardware‑aware scheduling, delivering 60‑85% speedups while preserving model quality.

DeepSeekGPU optimizationLLM inference
0 likes · 12 min read
DSpark Explained in 10 Essential Concepts: System‑Level Engineering Insights
Raymond Ops
Raymond Ops
Jun 27, 2026 · Artificial Intelligence

vLLM Quantized Inference: Loading AWQ/GPTQ Models and Optimizing GPU Memory

This article provides a step‑by‑step guide on using vLLM to load AWQ and GPTQ quantized large language models, covering environment setup, calibration data preparation, model quantization, deployment scripts, performance benchmarking, accuracy checks, best‑practice recommendations, and troubleshooting tips for GPU memory optimization.

AWQGPTQGPU memory optimization
0 likes · 45 min read
vLLM Quantized Inference: Loading AWQ/GPTQ Models and Optimizing GPU Memory
DeepHub IMBA
DeepHub IMBA
Jun 22, 2026 · Artificial Intelligence

How DiffusionGemma Shifts LLM Inference Bottleneck from Memory Bandwidth to Compute

DiffusionGemma, an experimental discrete text diffusion model built on the 26B MoE Gemma‑4 architecture, generates whole 256‑token blocks with bidirectional attention, moving the inference bottleneck from memory bandwidth to GPU compute, achieving up to four‑fold speed gains on H100 and RTX 5090 GPUs, though with lower output quality than standard autoregressive models.

DiffusionGemmaGPU performanceLLM inference
0 likes · 7 min read
How DiffusionGemma Shifts LLM Inference Bottleneck from Memory Bandwidth to Compute
Raymond Ops
Raymond Ops
Jun 22, 2026 · Artificial Intelligence

Elastic Deployment and GPU Scheduling for Large‑Model Inference with vLLM on Kubernetes

This article presents a detailed, step‑by‑step analysis of deploying the high‑performance vLLM inference engine on Kubernetes, covering GPU memory management, tensor parallelism, quantization choices, continuous batching, and automated scaling with HPA/KEDA to achieve low latency and high throughput for large language models.

DockerGPU schedulingKubernetes
0 likes · 49 min read
Elastic Deployment and GPU Scheduling for Large‑Model Inference with vLLM on Kubernetes
DataFunSummit
DataFunSummit
Jun 21, 2026 · Artificial Intelligence

Unified Scheduling Optimization for xLLM in Complex Business Scenarios

This article analyzes how the xLLM open‑source LLM inference engine tackles the coexistence of multiple priority levels and strict SLO latency targets by introducing a dynamic, SLO‑aware batch scheduler and a PD‑separation architecture that improve throughput and SLO satisfaction across diverse workloads.

Hierarchical Block ManagerKV CacheLLM inference
0 likes · 13 min read
Unified Scheduling Optimization for xLLM in Complex Business Scenarios
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 15, 2026 · Artificial Intelligence

vLLM 0.23.0 Brings Faster Local LLM Deployment and Wider Hardware Support

Version 0.23.0 of the open‑source vLLM inference engine adds full DeepSeek‑V4 stability, Model Runner V2 coverage for Llama, Mistral, Qwen3 and new models, a production‑grade Rust front‑end, multi‑level KV‑cache offloading, extensive hardware optimizations across NVIDIA, AMD, Intel, TPU and RISC‑V, plus API enhancements, delivering up to 20 % performance gains while simplifying deployment.

KV cache offloadingLLM inferenceRust front-end
0 likes · 8 min read
vLLM 0.23.0 Brings Faster Local LLM Deployment and Wider Hardware Support
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Jun 10, 2026 · Artificial Intelligence

LU‑KV Sets New SOTA at ICML 2026 by Redefining KV Cache Eviction

A joint effort by Baidu Baige and Fudan University introduces the LU‑KV framework, which treats KV‑cache budget allocation as a global combinatorial optimization problem, achieving only 0.52% relative performance loss at 80% compression and establishing a new efficiency‑accuracy SOTA on LongBench.

Cache EvictionEfficiencyICML 2026
0 likes · 5 min read
LU‑KV Sets New SOTA at ICML 2026 by Redefining KV Cache Eviction
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 7, 2026 · Artificial Intelligence

Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained

The article explains why LLM inference is memory‑bound, introduces vLLM’s three core optimizations—Continuous Batching, PagedAttention, and Prefix Caching—shows how to launch a vLLM server, run Python code to benchmark performance, and examines KV‑Cache memory usage with concrete numbers.

KV CacheLLM inferencePagedAttention
0 likes · 11 min read
Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained
Data Party THU
Data Party THU
Jun 6, 2026 · Artificial Intelligence

How a 400B MoE Model Runs on iPhone 17 Pro with Flash‑MoE

The article details how the open‑source Flash‑MoE engine enables the 400B‑parameter Qwen3.5‑397B‑A17B mixture‑of‑experts model to run on an iPhone 17 Pro, achieving about 0.6 tokens per second through a custom Metal pipeline, GCD‑driven SSD streaming, and aggressive caching strategies.

400BFlash-MoELLM inference
0 likes · 6 min read
How a 400B MoE Model Runs on iPhone 17 Pro with Flash‑MoE
Machine Heart
Machine Heart
Jun 5, 2026 · Artificial Intelligence

Stem Sparse Attention Cuts First-Token Latency by 3.6× for Long-Context LLMs

The article introduces Tencent Hunyuan's Stem sparse‑attention algorithm, which reduces first‑token latency by 3.6× on 128K context LLMs by reallocating compute with Token Position Decay and Output‑Aware Metric, and validates the gains with HPC‑optimized operators that outperform existing sparse methods in extensive benchmarks.

HPC OperatorsLLM inferenceOutput-Aware Metric
0 likes · 11 min read
Stem Sparse Attention Cuts First-Token Latency by 3.6× for Long-Context LLMs
Machine Heart
Machine Heart
May 31, 2026 · Artificial Intelligence

Can Low-Bit Models Cut Inference Costs Better Than Small Models?

The article analyzes how low‑bit quantization differs from simply using smaller LLMs, examines hardware‑level precision reduction, compares post‑training quantization with native low‑bit designs, and explains the runtime and testing requirements needed to achieve real inference cost savings.

Cost OptimizationLLM inferenceLow-Bit Quantization
0 likes · 7 min read
Can Low-Bit Models Cut Inference Costs Better Than Small Models?
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 29, 2026 · Artificial Intelligence

RTPurbo: >97% Sparsity and 9× Faster Long-Context LLM Inference with Minimal Training

The article presents RTPurbo, a lightweight two‑stage training method that converts full‑attention LLMs into highly sparse models with over 97% sparsity, achieving up to 9.36× prefill and 2.01× decode speedups while preserving near‑lossless accuracy across long‑context benchmarks up to 512K tokens.

Dynamic Token SelectionLLM inferenceLow-Dimensional Projection
0 likes · 17 min read
RTPurbo: >97% Sparsity and 9× Faster Long-Context LLM Inference with Minimal Training
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
May 27, 2026 · Artificial Intelligence

Optimizing Large Model Inference Architecture for the Agent Era: Engineering Practices and Challenges

The article analyzes the architectural challenges of large‑model inference in the Agent era—such as memory‑intensive MLA structures, MoE communication overhead, exploding KV‑Cache size, and tool‑call accuracy—and presents a series of engineering solutions including hierarchical KV‑Cache pooling, sequence parallelism, offloading strategies, and chip‑level adaptations to achieve higher throughput and lower token costs.

AI InfraAgentDeepSeek
0 likes · 15 min read
Optimizing Large Model Inference Architecture for the Agent Era: Engineering Practices and Challenges
Tencent Technical Engineering
Tencent Technical Engineering
May 25, 2026 · Artificial Intelligence

vLLM Deep Dive: Continuous Batching and Paged Attention for Fast LLM Inference

This article walks through a two‑month source‑code study of vLLM, explaining how token‑level scheduling, continuous batching, and the Paged Attention mechanism reshape tensor dimensions to turn large‑model inference into a compute‑bound, high‑throughput process while managing GPU memory efficiently.

FlashAttentionGPU optimizationLLM inference
0 likes · 29 min read
vLLM Deep Dive: Continuous Batching and Paged Attention for Fast LLM Inference
Old Zhang's AI Learning
Old Zhang's AI Learning
May 17, 2026 · Artificial Intelligence

Why DeepSeek V4 Flash’s Quantized Model Is Gaining Traction

The DeepSeek V4 Flash quantized GGUF model and the dedicated ds4 inference engine, both released by antirez, offer dramatically reduced activation parameters, massive 1‑million‑token context windows, aggressive KV‑cache compression and hardware‑specific quantizations that enable smooth local inference on high‑memory Macs and CUDA machines, while sacrificing generality for performance.

DS4DeepSeek-V4-FlashGGUF
0 likes · 11 min read
Why DeepSeek V4 Flash’s Quantized Model Is Gaining Traction
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
May 15, 2026 · Artificial Intelligence

How PD (Prefill‑Decode) Disaggregation Makes LLM Inference Faster and More Stable

The article explains PD (Prefill‑Decode) disaggregation, an architecture that separates the compute‑bound Prefill stage from the memory‑bound Decode stage onto different GPU pools, eliminating interference, enabling independent scaling, leveraging hardware specialization, and delivering up to 85% lower tail latency for large language model inference.

GPU scalingKV cache transportLLM inference
0 likes · 10 min read
How PD (Prefill‑Decode) Disaggregation Makes LLM Inference Faster and More Stable
Old Zhang's AI Learning
Old Zhang's AI Learning
May 13, 2026 · Artificial Intelligence

Why vLLM Now Leads Open‑Source LLM Inference Benchmarks

vLLM tops the Artificial Analysis ranking by delivering the highest throughput for DeepSeek V3.2, Qwen 3.5 397B, and MiniMax‑M2.5 on identical NVIDIA Blackwell Ultra hardware, thanks to extensive kernel‑fusion optimizations that remain in the main branch.

DeepSeekLLM inferenceQwen
0 likes · 7 min read
Why vLLM Now Leads Open‑Source LLM Inference Benchmarks
Huawei Cloud Developer Alliance
Huawei Cloud Developer Alliance
May 13, 2026 · Cloud Native

Why HPA Falls Short for LLMs and How Kthena Autoscaler Redefines Elastic Scaling

The article explains why traditional Kubernetes HPA cannot meet the unique demands of large‑language‑model inference, introduces Kthena Autoscaler’s model‑aware architecture, its dual stable/panic scaling modes, cost‑aware algorithms, flexible policy bindings, and provides practical configuration and observability guidance.

Kthena AutoscalerKubernetesLLM inference
0 likes · 10 min read
Why HPA Falls Short for LLMs and How Kthena Autoscaler Redefines Elastic Scaling
Lao Guo's Learning Space
Lao Guo's Learning Space
May 12, 2026 · Artificial Intelligence

Which Inference Framework Maximizes Your GPU Performance in 2026?

This article compares six popular LLM inference frameworks—vLLM, TensorRT‑LLM, llama.cpp, ds4.c, Ollama, and Omlx—across performance, ease of use, and hardware compatibility, then provides a practical matrix to help users select the best fit for their GPU.

Apple SiliconGPU performanceLLM inference
0 likes · 10 min read
Which Inference Framework Maximizes Your GPU Performance in 2026?
Machine Heart
Machine Heart
May 7, 2026 · Artificial Intelligence

Nvidia Endorses TokenSpeed: A Light‑Speed Agent Inference Engine Built in Two Months

TokenSpeed, an open‑source LLM inference engine designed for agent workloads, delivers TensorRT‑LLM‑level performance and vLLM‑level ease of use, outperforms TensorRT‑LLM by up to 11% throughput and halves latency on speculative decoding, and has earned Nvidia’s public recommendation.

LLM inferenceNVIDIA BlackwellTensorRT-LLM
0 likes · 8 min read
Nvidia Endorses TokenSpeed: A Light‑Speed Agent Inference Engine Built in Two Months
Old Zhang's AI Learning
Old Zhang's AI Learning
May 6, 2026 · Artificial Intelligence

Google Boosts Gemma 4 Inference Speed Up to 3× with MTP Drafter and Day‑0 vLLM Support

Google’s new Multi‑Token Prediction (MTP) drafter for Gemma 4 delivers up to three‑fold inference speedups across hardware and frameworks—validated by official benchmarks and independent DGX Spark tests—while preserving identical output quality, and is immediately usable via Hugging Face, vLLM, MLX, Ollama and edge‑device runtimes.

Apple SiliconGemma 4LLM inference
0 likes · 9 min read
Google Boosts Gemma 4 Inference Speed Up to 3× with MTP Drafter and Day‑0 vLLM Support
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 21, 2026 · Artificial Intelligence

Prefill-as-a-Service Boosts LLM Inference Throughput by 54%

A joint Moonshot AI and Tsinghua study shows that the Prefill-as-a-Service (PrfaaS) architecture, enabled by hybrid‑attention models that shrink KVCache size, can offload long Prefill work to a remote cluster and, with dual‑timescale scheduling, achieve a 54% throughput gain over homogeneous PD deployment and 32% over naive heterogeneous setups.

KVCache optimizationLLM inferencePrefill-as-a-Service
0 likes · 12 min read
Prefill-as-a-Service Boosts LLM Inference Throughput by 54%
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 11, 2026 · Artificial Intelligence

Mastering SGLang: KV Cache and RadixAttention for Faster LLM Inference

This article reviews the DeepLearning.ai short course on SGLang, explains why large‑language‑model inference is slow, details how KV Cache reduces the computation from O(n²) to O(n), introduces RadixAttention for cross‑request caching, and presents code examples and benchmark results showing up to 10× speedup in real‑world RAG scenarios.

KV CacheLLM inferencePython
0 likes · 13 min read
Mastering SGLang: KV Cache and RadixAttention for Faster LLM Inference
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 7, 2026 · Artificial Intelligence

vLLM 0.19.0: HuggingFace v5 Support, Multimodal Boosts, and CPU KV Cache Offload

The vLLM 0.19.0 release adds first‑day Gemma 4 support, merges zero‑bubble asynchronous scheduling with speculative decoding, matures Model Runner V2, introduces full‑CUDA‑graph acceleration for ViT, generalizes DBO, brings CPU KV cache offload, and expands hardware and Transformers compatibility, offering substantial performance and flexibility gains for production LLM inference.

CPU KV offloadGPUGemma 4
0 likes · 18 min read
vLLM 0.19.0: HuggingFace v5 Support, Multimodal Boosts, and CPU KV Cache Offload