BestHub
Sign in
Tagged articles

LLM inference

107 articles · Page 1 of 2
Ops Development & AI Practice
Ops Development & AI Practice
Sep 26, 2026 · Artificial Intelligence

Mac mini M6 32GB LLM Inference: Real TPS, Bandwidth Limits & Hybrid Cloud Strategy

This analysis dissects the 32GB Mac mini M6's true LLM inference capabilities, revealing 170 GB/s unified memory bandwidth limits, GPU compute constraints versus AMD Strix Halo, measured tokens-per-second for 8B–32B models, and a cost-driven argument for hybrid local-cloud deployment over expensive local-only workstations.

Apple SiliconLLM inferenceM6 chip
0 likes · 21 min read
Mac mini M6 32GB LLM Inference: Real TPS, Bandwidth Limits & Hybrid Cloud Strategy
Ops Development & AI Practice
Ops Development & AI Practice
Sep 26, 2026 · Artificial Intelligence

Local LLM Hardware Truths: MoE, CPU/GPU/NPU, and 32GB Mac Mini Tuning

This article explains why CPUs struggle with MoE model prefill but excel at decode, compares dense, SSM, MLA, and BitNet architectures, evaluates CPU, NVIDIA GPU, Apple UMA, and NPU hardware for local LLM inference, and provides a practical tuning guide for running 32B models on a 32GB Mac mini with MLX and Ollama.

Apple SiliconKV CacheLLM inference
0 likes · 21 min read
Local LLM Hardware Truths: MoE, CPU/GPU/NPU, and 32GB Mac Mini Tuning
Lao Guo's Learning Space
Lao Guo's Learning Space
Sep 23, 2026 · Artificial Intelligence

Mac mini 16GB to Studio 256GB: Which Mac Runs Your LLMs Best?

This article analyzes Mac mini and Mac Studio configurations for local LLM inference, showing how unified memory capacity determines which models fit and memory bandwidth dictates token generation speed, with real-world benchmarks for 8B to 235B models across M6, M5 Pro, M5 Max, and M5 Ultra chips.

Apple SiliconLLM inferenceMac Studio
0 likes · 13 min read
Mac mini 16GB to Studio 256GB: Which Mac Runs Your LLMs Best?
Lao Guo's Learning Space
Lao Guo's Learning Space
Sep 23, 2026 · Artificial Intelligence

M5 Ultra 256GB vs RTX 5090 32GB: Local LLM Inference Compared

This article compares Apple M5 Ultra (256GB unified memory) and NVIDIA RTX 5090 (32GB GDDR7) for local LLM inference across 27B–100B+ models, measuring capacity limits, token throughput, power draw, and CUDA vs MLX ecosystems, concluding M5 Ultra excels beyond 32GB while RTX 5090 leads on smaller models.

CUDALLM inferenceM5 Ultra
0 likes · 10 min read
M5 Ultra 256GB vs RTX 5090 32GB: Local LLM Inference Compared
Lao Guo's Learning Space
Lao Guo's Learning Space
Sep 22, 2026 · Industry Insights

M5 Ultra 256GB Benchmarks: 27B LLM 51 tok/s, GPU +82% in Cyberpunk 4K RT

Hands-on benchmarks of the M5 Ultra 256GB Mac Studio show 45.7% faster single-stream LLM decoding, 3.6x higher multi-user prefill throughput, 43.6% CPU multi-core gains, 67.8% GPU improvement, and 82.8% higher Cyberpunk 2077 4K ray-tracing frame rates versus M3 Ultra, with purchasing guidance.

BenchmarkGPU performanceLLM inference
0 likes · 13 min read
M5 Ultra 256GB Benchmarks: 27B LLM 51 tok/s, GPU +82% in Cyberpunk 4K RT
Lao Guo's Learning Space
Lao Guo's Learning Space
Sep 22, 2026 · Industry Insights

M5 Ultra 64 vs 80 Core GPU: What ¥9750 Buys – Not Faster LLM Decoding

The article compares Apple M5 Ultra 64-core and 80-core GPU variants, revealing that the ¥9750 price difference adds 16 GPU cores, 6 CPU cores, and 512GB memory support, but identical 1.2TB/s memory bandwidth means LLM decoding speed remains unchanged; the upgrade only benefits GPU-intensive tasks like 3D rendering and video effects.

3D renderingApple M5 UltraGPU cores
0 likes · 10 min read
M5 Ultra 64 vs 80 Core GPU: What ¥9750 Buys – Not Faster LLM Decoding
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Sep 21, 2026 · Artificial Intelligence

Random KV Cache Eviction Rivals Top Baselines, Boosts Throughput 43%

Salesforce AI Research and UIUC propose Random Attention, a simple KV cache eviction method that randomly retains reasoning tokens while protecting the prompt, matching state-of-the-art baselines across math, science, and code reasoning tasks and improving inference throughput by 32–43% in vLLM serving.

Cache EvictionKV CacheLLM inference
0 likes · 20 min read
Random KV Cache Eviction Rivals Top Baselines, Boosts Throughput 43%
Lao Guo's Learning Space
Lao Guo's Learning Space
Sep 21, 2026 · Artificial Intelligence

M5 Ultra 256GB: Which LLMs Fit? 27B to 200B+ Capacity Analysis

The article analyzes Apple M5 Ultra 256GB unified memory capacity for local LLM inference, showing Q4-quantized models from 27B to 200B+ fit, but KV cache overhead for long contexts reduces headroom; it compares with multi-GPU setups and advises on purchase decisions.

Apple SiliconHardware AnalysisKV Cache
0 likes · 8 min read
M5 Ultra 256GB: Which LLMs Fit? 27B to 200B+ Capacity Analysis
Machine Heart
Machine Heart
Sep 19, 2026 · Artificial Intelligence

Random Attention: Random KV Cache Eviction Rivals Best Baselines, Boosts Throughput 43%

Salesforce and UIUC researchers propose Random Attention, a simple KV cache eviction method that randomly retains reasoning tokens while fully protecting the prompt, matching the accuracy of complex importance-based methods across multiple models and tasks while increasing inference throughput by 32-43% in vLLM serving.

Attention MechanismCache CompressionEfficient AI
0 likes · 18 min read
Random Attention: Random KV Cache Eviction Rivals Best Baselines, Boosts Throughput 43%
Architects' Tech Alliance
Architects' Tech Alliance
Sep 5, 2026 · Artificial Intelligence

CloudMatrix384: Co-Designing Supernodes for Trillion-Parameter LLM Inference

Huawei's CloudMatrix384 integrates 384 Ascend 910 NPUs with a unified bus network and decoupled PDC service architecture to achieve 4.45 tokens/s/TFLOPS prefill efficiency on DeepSeek-R1 671B, outperforming H100 baselines through fused MoE communication, INT8 quantization, and heterogeneous pipelining.

Ascend 910CloudMatrix384Expert Parallelism
0 likes · 20 min read
CloudMatrix384: Co-Designing Supernodes for Trillion-Parameter LLM Inference
DeepHub IMBA
DeepHub IMBA
Sep 3, 2026 · Artificial Intelligence

FlashSpec: Adaptive LLM Inference with Speculative Decoding — Six Hard-Won Lessons

FlashSpec implements speculative decoding using GPU-native Triton kernel verification and online bandit-based draft model selection, sharing six practical lessons on specification-first development, hidden temperature bugs, cross-platform packaging pitfalls, kernel performance trade-offs, property-based testing value, and adaptive algorithm prerequisites.

CI/CDLLM inferenceThompson sampling
0 likes · 15 min read
FlashSpec: Adaptive LLM Inference with Speculative Decoding — Six Hard-Won Lessons
AI Engineering
AI Engineering
Aug 28, 2026 · Cloud Native

Why Round‑Robin Fails for LLM Inference and How llm‑d Fixes It

Round‑robin routing in Kubernetes wipes out KV‑cache benefits for LLM inference, but llm‑d introduces cache‑aware routing, hierarchical eviction, and prefill/decode separation, delivering up to three‑fold throughput gains and halving first‑token latency, as shown in Tesla's production rollout.

KV CacheKubernetesLLM inference
0 likes · 7 min read
Why Round‑Robin Fails for LLM Inference and How llm‑d Fixes It
ThinkingAgent
ThinkingAgent
Aug 24, 2026 · Artificial Intelligence

Why Quantization and KV‑Cache Are Key to High‑Performance LLM Inference

The article analyzes why the same LLM can exhibit vastly different cost, speed, and concurrency across inference systems, showing that KV‑cache memory management, continuous batching, PagedAttention, quantization trade‑offs, and speculative decoding together determine real‑world throughput and latency.

KV CacheLLM inferencePerformance Optimization
0 likes · 39 min read
Why Quantization and KV‑Cache Are Key to High‑Performance LLM Inference
AI Architecture Path
AI Architecture Path
Aug 7, 2026 · Artificial Intelligence

Run 2.8‑Trillion‑Parameter Models on a 4 GB GPU with AirLLM’s Lossless Inference

AirLLM lets ordinary 4‑GB consumer GPUs run massive LLMs such as 70B Llama and the 2.8‑trillion‑parameter Kimi K3 MoE without quantisation, by streaming model layers from disk, offering a detailed comparison with traditional low‑VRAM tricks, step‑by‑step installation, pitfalls, scenario‑based guidance and an assessment of strengths and risks.

AirLLMLLM inferenceMOE
0 likes · 16 min read
Run 2.8‑Trillion‑Parameter Models on a 4 GB GPU with AirLLM’s Lossless Inference
DeepHub IMBA
DeepHub IMBA
Aug 3, 2026 · Artificial Intelligence

How Continuous Batching Boosts LLM Throughput by Up to 36×

The article explains continuous batching, a scheduling technique that re‑evaluates the batch at every inference iteration, eliminating the straggler problem of static batching, and shows how it, together with PagedAttention, can raise LLM service throughput dramatically while balancing latency and memory usage.

GPU utilizationLLM inferencecontinuous batching
0 likes · 9 min read
How Continuous Batching Boosts LLM Throughput by Up to 36×
AI Engineer Programming
AI Engineer Programming
Aug 2, 2026 · Artificial Intelligence

Comprehensive Cost Assessment of End-to-End RAG Systems

This report breaks down production‑grade Retrieval‑Augmented Generation (RAG) system costs into five modules—LLM inference, vector database, embedding, bandwidth, and infrastructure—revealing that model choice drives over 40% of expenses, quantisation can halve vector costs, and multimodal storage may outpace vector database spending.

LLM inferenceMultimodalRAG
0 likes · 14 min read
Comprehensive Cost Assessment of End-to-End RAG Systems
MaGe Linux Operations
MaGe Linux Operations
Jul 21, 2026 · Cloud Native

Auto‑Scaling LLM Inference with Kubernetes HPA Based on Request Queue Depth

The article explains how to replace CPU‑only autoscaling for large‑model inference services with a Kubernetes HPA that scales pods according to a custom queue‑depth metric exported to Prometheus, covering metric definition, deployment configuration, Prometheus‑Adapter setup, HPA creation, capacity calculation, validation, troubleshooting, and rollback procedures.

AutoscalingHPAKubernetes
0 likes · 21 min read
Auto‑Scaling LLM Inference with Kubernetes HPA Based on Request Queue Depth
Machine Heart
Machine Heart
Jul 20, 2026 · Artificial Intelligence

Li Xiuhong on Cross-Cluster Heterogeneous PD Separation and Token Factory “Super Pipeline” at WAIC

The article details how 无问芯穹’s Agentic Infra strategy uses a cross‑cluster heterogeneous PD‑separation architecture (PDD) to cut first‑token latency by 51.5% and token cost by 37.5%, explains the bandwidth bottleneck of KV‑Cache transfer, introduces Decode‑side RadixCache and three‑stage handoff mechanisms, and shows a 37.5% BCR improvement that translates into roughly ten‑fold inference cost reduction.

Agentic InfraCross-ClusterLLM inference
0 likes · 25 min read
Li Xiuhong on Cross-Cluster Heterogeneous PD Separation and Token Factory “Super Pipeline” at WAIC
MaGe Linux Operations
MaGe Linux Operations
Jul 19, 2026 · Operations

How to Deploy vLLM for an OpenAI‑Compatible Inference Service

This guide walks through deploying vLLM on Linux with NVIDIA GPUs and Docker Compose, covering service boundaries, host and container checks, model directory validation, image and parameter verification, minimal startup, Compose configuration, API testing, concurrency tuning, multi‑GPU troubleshooting, Nginx exposure, upgrade/rollback procedures, and daily health checks.

Docker ComposeGPUKV Cache
0 likes · 18 min read
How to Deploy vLLM for an OpenAI‑Compatible Inference Service
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
Jul 16, 2026 · Artificial Intelligence

Cut First‑Token Latency by 3.25×: Introducing HYPIC for Position‑Independent Caching in Hybrid‑Attention LLMs

HYPIC combines position‑independent caching with hybrid‑attention LLMs, reducing first‑token latency by 3.25× and sustaining QPS by 1.66× while keeping quality loss under 2 points, through a segment‑cumulative transition operator, a seam‑window fix for full‑attention layers, and segment‑parallel execution.

HYPICKV CacheLLM inference
0 likes · 13 min read
Cut First‑Token Latency by 3.25×: Introducing HYPIC for Position‑Independent Caching in Hybrid‑Attention LLMs
Machine Heart
Machine Heart
Jul 16, 2026 · Artificial Intelligence

Brake Overthinking in Long‑Reasoning Models by Detecting Semantic Redundancy

Long‑thinking LLMs often waste 41‑52% of tokens after the final answer; the PUMA framework detects when reasoning stops producing new semantic information, enabling early exit that cuts average token usage by 26.2% while keeping accuracy stable and even improving speed across multiple benchmarks.

LLM inferencePUMAearly exit
0 likes · 9 min read
Brake Overthinking in Long‑Reasoning Models by Detecting Semantic Redundancy
AI Engineer Programming
AI Engineer Programming
Jul 14, 2026 · Backend Development

How vLLM Generates Tokens: A Deep Dive into the Source Code

This article walks through vLLM’s token‑generation pipeline by dissecting its source files, process roles, ZMQ communication, scheduling, and deployment considerations, revealing how prompts become streamed token IDs and highlighting the key factors that limit concurrency and performance.

GPU memoryLLM inferenceZMQ
0 likes · 17 min read
How vLLM Generates Tokens: A Deep Dive into the Source Code
Alimama Tech
Alimama Tech
Jul 8, 2026 · Artificial Intelligence

Optimizing Qwen Series Inference with SGLang‑ATOM on AMD Instinct GPUs

Alibaba Mama AI Infra and AMD collaborated to integrate SGLang‑ATOM plugin mode for Qwen 3.5/3.6 models on MI308X GPUs, detailing multimodal, GDN, KV‑cache, and parallel‑strategy optimizations that achieve low‑latency, high‑throughput inference suitable for complex advertising workloads.

AMD ATOMGPU optimizationLLM inference
0 likes · 18 min read
Optimizing Qwen Series Inference with SGLang‑ATOM on AMD Instinct GPUs
Data Party THU
Data Party THU
Jul 7, 2026 · Artificial Intelligence

Parallel Decoding for Large Language Models: Balancing Inference Speed and Sampling Diversity in E‑GRM

The article presents an engineering analysis of the E‑GRM framework, detailing how parallel decoding, a temperature‑ladder sampling strategy, and batch‑parallel KV‑Cache sharing achieve low‑latency, high‑diversity inference while preserving consensus‑driven routing accuracy.

Batch ParallelismConsensus RoutingE‑GRM
0 likes · 14 min read
Parallel Decoding for Large Language Models: Balancing Inference Speed and Sampling Diversity in E‑GRM
AI Tech Publishing
AI Tech Publishing
Jun 29, 2026 · Artificial Intelligence

Understanding the Full LLM Inference Pipeline: From Tokenization to Streaming Output

The article walks through every stage of LLM inference—from tokenization and embedding, through transformer layers, prefill (compute‑bound) and decode (memory‑bound) phases, KV‑cache management, attention redesign, quantization, and serving infrastructure—explaining how each step consumes time and resources and how to identify bottlenecks.

Attention optimizationKV CacheLLM inference
0 likes · 14 min read
Understanding the Full LLM Inference Pipeline: From Tokenization to Streaming Output
DataFunTalk
DataFunTalk
Jun 29, 2026 · Artificial Intelligence

DSpark Explained: 10 Key Concepts You Need to Know

The DSpark system from DeepSeek combines batch decoding, speculative decoding, draft‑model tricks, Eagle‑MTP, DFlash parallelism, variable‑length scheduling and online confidence calibration to deliver up to 85% speedup and four‑fold throughput gains while maintaining generation quality.

Batch DecodingDFlashDSpark
0 likes · 12 min read
DSpark Explained: 10 Key Concepts You Need to Know
PaperAgent
PaperAgent
Jun 29, 2026 · Artificial Intelligence

DeepSeek Opens DSpark: A New Speculative Decoding Framework for Large Language Models

DeepSeek releases DSpark, an open‑source speculative decoding system that combines semi‑autoregressive generation with confidence‑scheduled verification, delivering 60‑85% per‑user speed gains, lower latency, and superior acceptance rates compared with Eagle3 and DFlash across multiple LLM benchmarks.

Confidence SchedulingLLM inferenceSemi-Autoregressive Generation
0 likes · 14 min read
DeepSeek Opens DSpark: A New Speculative Decoding Framework for Large Language Models
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jun 28, 2026 · Artificial Intelligence

DSpark Explained in 10 Essential Concepts: System‑Level Engineering Insights

DSpark, DeepSeek’s new LLM inference framework, combines batch processing, speculative decoding, Eagle‑style draft models and DFlash‑style parallel generation with a lightweight sequential head and hardware‑aware scheduling, delivering 60‑85% speedups while preserving model quality.

Batch ProcessingDeepSeekGPU optimization
0 likes · 12 min read
DSpark Explained in 10 Essential Concepts: System‑Level Engineering Insights
Raymond Ops
Raymond Ops
Jun 27, 2026 · Artificial Intelligence

vLLM Quantized Inference: Loading AWQ/GPTQ Models and Optimizing GPU Memory

This article provides a step‑by‑step guide on using vLLM to load AWQ and GPTQ quantized large language models, covering environment setup, calibration data preparation, model quantization, deployment scripts, performance benchmarking, accuracy checks, best‑practice recommendations, and troubleshooting tips for GPU memory optimization.

AWQGPTQGPU memory optimization
0 likes · 45 min read
vLLM Quantized Inference: Loading AWQ/GPTQ Models and Optimizing GPU Memory
DeepHub IMBA
DeepHub IMBA
Jun 22, 2026 · Artificial Intelligence

How DiffusionGemma Shifts LLM Inference Bottleneck from Memory Bandwidth to Compute

DiffusionGemma, an experimental discrete text diffusion model built on the 26B MoE Gemma‑4 architecture, generates whole 256‑token blocks with bidirectional attention, moving the inference bottleneck from memory bandwidth to GPU compute, achieving up to four‑fold speed gains on H100 and RTX 5090 GPUs, though with lower output quality than standard autoregressive models.

DiffusionGemmaGPU performanceLLM inference
0 likes · 7 min read
How DiffusionGemma Shifts LLM Inference Bottleneck from Memory Bandwidth to Compute
Raymond Ops
Raymond Ops
Jun 22, 2026 · Artificial Intelligence

Elastic Deployment and GPU Scheduling for Large‑Model Inference with vLLM on Kubernetes

This article presents a detailed, step‑by‑step analysis of deploying the high‑performance vLLM inference engine on Kubernetes, covering GPU memory management, tensor parallelism, quantization choices, continuous batching, and automated scaling with HPA/KEDA to achieve low latency and high throughput for large language models.

DockerGPU SchedulingKubernetes
0 likes · 49 min read
Elastic Deployment and GPU Scheduling for Large‑Model Inference with vLLM on Kubernetes
DataFunSummit
DataFunSummit
Jun 21, 2026 · Artificial Intelligence

Unified Scheduling Optimization for xLLM in Complex Business Scenarios

This article analyzes how the xLLM open‑source LLM inference engine tackles the coexistence of multiple priority levels and strict SLO latency targets by introducing a dynamic, SLO‑aware batch scheduler and a PD‑separation architecture that improve throughput and SLO satisfaction across diverse workloads.

Hierarchical Block ManagerKV CacheLLM inference
0 likes · 13 min read
Unified Scheduling Optimization for xLLM in Complex Business Scenarios
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 15, 2026 · Artificial Intelligence

vLLM 0.23.0 Brings Faster Local LLM Deployment and Wider Hardware Support

Version 0.23.0 of the open‑source vLLM inference engine adds full DeepSeek‑V4 stability, Model Runner V2 coverage for Llama, Mistral, Qwen3 and new models, a production‑grade Rust front‑end, multi‑level KV‑cache offloading, extensive hardware optimizations across NVIDIA, AMD, Intel, TPU and RISC‑V, plus API enhancements, delivering up to 20 % performance gains while simplifying deployment.

DeepSeek-V4KV cache offloadingLLM inference
0 likes · 8 min read
vLLM 0.23.0 Brings Faster Local LLM Deployment and Wider Hardware Support
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Jun 10, 2026 · Artificial Intelligence

LU‑KV Sets New SOTA at ICML 2026 by Redefining KV Cache Eviction

A joint effort by Baidu Baige and Fudan University introduces the LU‑KV framework, which treats KV‑cache budget allocation as a global combinatorial optimization problem, achieving only 0.52% relative performance loss at 80% compression and establishing a new efficiency‑accuracy SOTA on LongBench.

Cache EvictionICML 2026KV Cache
0 likes · 5 min read
LU‑KV Sets New SOTA at ICML 2026 by Redefining KV Cache Eviction
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 7, 2026 · Artificial Intelligence

Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained

The article explains why LLM inference is memory‑bound, introduces vLLM’s three core optimizations—Continuous Batching, PagedAttention, and Prefix Caching—shows how to launch a vLLM server, run Python code to benchmark performance, and examines KV‑Cache memory usage with concrete numbers.

KV CacheLLM inferencePagedAttention
0 likes · 11 min read
Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained
Data Party THU
Data Party THU
Jun 6, 2026 · Artificial Intelligence

How a 400B MoE Model Runs on iPhone 17 Pro with Flash‑MoE

The article details how the open‑source Flash‑MoE engine enables the 400B‑parameter Qwen3.5‑397B‑A17B mixture‑of‑experts model to run on an iPhone 17 Pro, achieving about 0.6 tokens per second through a custom Metal pipeline, GCD‑driven SSD streaming, and aggressive caching strategies.

400BFlash-MoELLM inference
0 likes · 6 min read
How a 400B MoE Model Runs on iPhone 17 Pro with Flash‑MoE
Machine Heart
Machine Heart
Jun 5, 2026 · Artificial Intelligence

Stem Sparse Attention Cuts First-Token Latency by 3.6× for Long-Context LLMs

The article introduces Tencent Hunyuan's Stem sparse‑attention algorithm, which reduces first‑token latency by 3.6× on 128K context LLMs by reallocating compute with Token Position Decay and Output‑Aware Metric, and validates the gains with HPC‑optimized operators that outperform existing sparse methods in extensive benchmarks.

HPC OperatorsLLM inferenceOutput-Aware Metric
0 likes · 11 min read
Stem Sparse Attention Cuts First-Token Latency by 3.6× for Long-Context LLMs
Machine Heart
Machine Heart
May 31, 2026 · Artificial Intelligence

Can Low-Bit Models Cut Inference Costs Better Than Small Models?

The article analyzes how low‑bit quantization differs from simply using smaller LLMs, examines hardware‑level precision reduction, compares post‑training quantization with native low‑bit designs, and explains the runtime and testing requirements needed to achieve real inference cost savings.

Cost OptimizationLLM inferencehardware-acceleration
0 likes · 7 min read
Can Low-Bit Models Cut Inference Costs Better Than Small Models?
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 29, 2026 · Artificial Intelligence

RTPurbo: >97% Sparsity and 9× Faster Long-Context LLM Inference with Minimal Training

The article presents RTPurbo, a lightweight two‑stage training method that converts full‑attention LLMs into highly sparse models with over 97% sparsity, achieving up to 9.36× prefill and 2.01× decode speedups while preserving near‑lossless accuracy across long‑context benchmarks up to 512K tokens.

Dynamic Token SelectionLLM inferenceLow-Dimensional Projection
0 likes · 17 min read
RTPurbo: >97% Sparsity and 9× Faster Long-Context LLM Inference with Minimal Training
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
May 27, 2026 · Artificial Intelligence

Optimizing Large Model Inference Architecture for the Agent Era: Engineering Practices and Challenges

The article analyzes the architectural challenges of large‑model inference in the Agent era—such as memory‑intensive MLA structures, MoE communication overhead, exploding KV‑Cache size, and tool‑call accuracy—and presents a series of engineering solutions including hierarchical KV‑Cache pooling, sequence parallelism, offloading strategies, and chip‑level adaptations to achieve higher throughput and lower token costs.

AI InfraAgentDeepSeek
0 likes · 15 min read
Optimizing Large Model Inference Architecture for the Agent Era: Engineering Practices and Challenges
Tencent Technical Engineering
Tencent Technical Engineering
May 25, 2026 · Artificial Intelligence

vLLM Deep Dive: Continuous Batching and Paged Attention for Fast LLM Inference

This article walks through a two‑month source‑code study of vLLM, explaining how token‑level scheduling, continuous batching, and the Paged Attention mechanism reshape tensor dimensions to turn large‑model inference into a compute‑bound, high‑throughput process while managing GPU memory efficiently.

FlashAttentionGPU optimizationLLM inference
0 likes · 29 min read
vLLM Deep Dive: Continuous Batching and Paged Attention for Fast LLM Inference
Old Zhang's AI Learning
Old Zhang's AI Learning
May 17, 2026 · Artificial Intelligence

Why DeepSeek V4 Flash’s Quantized Model Is Gaining Traction

The DeepSeek V4 Flash quantized GGUF model and the dedicated ds4 inference engine, both released by antirez, offer dramatically reduced activation parameters, massive 1‑million‑token context windows, aggressive KV‑cache compression and hardware‑specific quantizations that enable smooth local inference on high‑memory Macs and CUDA machines, while sacrificing generality for performance.

DS4DeepSeek-V4-FlashGGUF
0 likes · 11 min read
Why DeepSeek V4 Flash’s Quantized Model Is Gaining Traction
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
May 15, 2026 · Artificial Intelligence

How PD (Prefill‑Decode) Disaggregation Makes LLM Inference Faster and More Stable

The article explains PD (Prefill‑Decode) disaggregation, an architecture that separates the compute‑bound Prefill stage from the memory‑bound Decode stage onto different GPU pools, eliminating interference, enabling independent scaling, leveraging hardware specialization, and delivering up to 85% lower tail latency for large language model inference.

GPU scalingKV cache transportLLM inference
0 likes · 10 min read
How PD (Prefill‑Decode) Disaggregation Makes LLM Inference Faster and More Stable
Old Zhang's AI Learning
Old Zhang's AI Learning
May 13, 2026 · Artificial Intelligence

Why vLLM Now Leads Open‑Source LLM Inference Benchmarks

vLLM tops the Artificial Analysis ranking by delivering the highest throughput for DeepSeek V3.2, Qwen 3.5 397B, and MiniMax‑M2.5 on identical NVIDIA Blackwell Ultra hardware, thanks to extensive kernel‑fusion optimizations that remain in the main branch.

DeepSeekLLM inferenceQwen
0 likes · 7 min read
Why vLLM Now Leads Open‑Source LLM Inference Benchmarks
Huawei Cloud Developer Alliance
Huawei Cloud Developer Alliance
May 13, 2026 · Cloud Native

Why HPA Falls Short for LLMs and How Kthena Autoscaler Redefines Elastic Scaling

The article explains why traditional Kubernetes HPA cannot meet the unique demands of large‑language‑model inference, introduces Kthena Autoscaler’s model‑aware architecture, its dual stable/panic scaling modes, cost‑aware algorithms, flexible policy bindings, and provides practical configuration and observability guidance.

AutoscalingKthena AutoscalerKubernetes
0 likes · 10 min read
Why HPA Falls Short for LLMs and How Kthena Autoscaler Redefines Elastic Scaling
Lao Guo's Learning Space
Lao Guo's Learning Space
May 12, 2026 · Artificial Intelligence

Which Inference Framework Maximizes Your GPU Performance in 2026?

This article compares six popular LLM inference frameworks—vLLM, TensorRT‑LLM, llama.cpp, ds4.c, Ollama, and Omlx—across performance, ease of use, and hardware compatibility, then provides a practical matrix to help users select the best fit for their GPU.

Apple SiliconGPU performanceLLM inference
0 likes · 10 min read
Which Inference Framework Maximizes Your GPU Performance in 2026?
Machine Heart
Machine Heart
May 7, 2026 · Artificial Intelligence

Nvidia Endorses TokenSpeed: A Light‑Speed Agent Inference Engine Built in Two Months

TokenSpeed, an open‑source LLM inference engine designed for agent workloads, delivers TensorRT‑LLM‑level performance and vLLM‑level ease of use, outperforms TensorRT‑LLM by up to 11% throughput and halves latency on speculative decoding, and has earned Nvidia’s public recommendation.

Agent workloadsLLM inferenceNVIDIA Blackwell
0 likes · 8 min read
Nvidia Endorses TokenSpeed: A Light‑Speed Agent Inference Engine Built in Two Months
Old Zhang's AI Learning
Old Zhang's AI Learning
May 6, 2026 · Artificial Intelligence

Google Boosts Gemma 4 Inference Speed Up to 3× with MTP Drafter and Day‑0 vLLM Support

Google’s new Multi‑Token Prediction (MTP) drafter for Gemma 4 delivers up to three‑fold inference speedups across hardware and frameworks—validated by official benchmarks and independent DGX Spark tests—while preserving identical output quality, and is immediately usable via Hugging Face, vLLM, MLX, Ollama and edge‑device runtimes.

Apple SiliconGemma 4LLM inference
0 likes · 9 min read
Google Boosts Gemma 4 Inference Speed Up to 3× with MTP Drafter and Day‑0 vLLM Support
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 21, 2026 · Artificial Intelligence

Prefill-as-a-Service Boosts LLM Inference Throughput by 54%

A joint Moonshot AI and Tsinghua study shows that the Prefill-as-a-Service (PrfaaS) architecture, enabled by hybrid‑attention models that shrink KVCache size, can offload long Prefill work to a remote cluster and, with dual‑timescale scheduling, achieve a 54% throughput gain over homogeneous PD deployment and 32% over naive heterogeneous setups.

KVCache optimizationLLM inferencePrefill-as-a-Service
0 likes · 12 min read
Prefill-as-a-Service Boosts LLM Inference Throughput by 54%
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 11, 2026 · Artificial Intelligence

Mastering SGLang: KV Cache and RadixAttention for Faster LLM Inference

This article reviews the DeepLearning.ai short course on SGLang, explains why large‑language‑model inference is slow, details how KV Cache reduces the computation from O(n²) to O(n), introduces RadixAttention for cross‑request caching, and presents code examples and benchmark results showing up to 10× speedup in real‑world RAG scenarios.

KV CacheLLM inferencePerformance Optimization
0 likes · 13 min read
Mastering SGLang: KV Cache and RadixAttention for Faster LLM Inference
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 7, 2026 · Artificial Intelligence

vLLM 0.19.0: HuggingFace v5 Support, Multimodal Boosts, and CPU KV Cache Offload

The vLLM 0.19.0 release adds first‑day Gemma 4 support, merges zero‑bubble asynchronous scheduling with speculative decoding, matures Model Runner V2, introduces full‑CUDA‑graph acceleration for ViT, generalizes DBO, brings CPU KV cache offload, and expands hardware and Transformers compatibility, offering substantial performance and flexibility gains for production LLM inference.

CPU KV offloadGPUGemma 4
0 likes · 18 min read
vLLM 0.19.0: HuggingFace v5 Support, Multimodal Boosts, and CPU KV Cache Offload
ShiZhen AI
ShiZhen AI
Apr 2, 2026 · Artificial Intelligence

How KV Cache Works and Why Large Model Outputs Cost Five Times More Than Inputs

The article explains the KV Cache mechanism that stores previously computed key/value vectors to avoid redundant Transformer calculations, delivering roughly a 5× speedup, while also detailing why generating output tokens is far more expensive than processing input tokens due to serial generation and memory trade‑offs.

KV CacheLLM inferenceMemory Optimization
0 likes · 9 min read
How KV Cache Works and Why Large Model Outputs Cost Five Times More Than Inputs
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Mar 23, 2026 · Artificial Intelligence

How vLLM‑Kunlun Unlocks Peak LLM Performance on Kunlun XPU

This article details the technical challenges of adapting the open‑source vLLM inference framework to Baidu's Kunlun XPU, outlines four major performance bottlenecks, and presents a multi‑dimensional optimization roadmap—including custom plugins, operator fusion, INT8 quantization, and CUDA‑Graph techniques—that together boost throughput by up to 8% and narrow the gap with leading GPU hardware.

CUDA GraphINT8 quantizationKunlun XPU
0 likes · 13 min read
How vLLM‑Kunlun Unlocks Peak LLM Performance on Kunlun XPU
Old Zhang's AI Learning
Old Zhang's AI Learning
Mar 18, 2026 · Artificial Intelligence

Running Claude‑Opus‑4.6‑Distilled Qwen3.5 27B on a Single RTX 4090 with llama.cpp: 46 tokens/s Performance

The article details a hands‑on test of the Claude‑Opus‑4.6‑distilled Qwen3.5 27B model running on a single RTX 4090 via llama.cpp, showing a steady 46 tokens per second generation speed, a 64K context window, and a step‑by‑step Docker‑based setup while comparing it to GLM‑4.7‑Flash‑AWQ‑4bit and discussing llama.cpp’s limitations for multi‑GPU inference.

Claude OpusDockerLLM inference
0 likes · 5 min read
Running Claude‑Opus‑4.6‑Distilled Qwen3.5 27B on a Single RTX 4090 with llama.cpp: 46 tokens/s Performance
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Mar 5, 2026 · Artificial Intelligence

Mamba’s SSD Framework Shatters Serial Bottleneck, Outperforms vLLM and SGLang

The new Speculative Speculative Decoding (SSD) framework, built by the Mamba and FlashAttention authors, eliminates the serial draft‑verification bottleneck in LLM inference by running the draft model asynchronously, introducing a speculation cache and the Saguaro algorithm, which together deliver up to 5× speedup over autoregressive baselines and up to 2× over optimized engines on Llama‑3 and Qwen‑3, reshaping the latency‑throughput trade‑off.

Asynchronous ParallelismLLM inferencePerformance Optimization
0 likes · 9 min read
Mamba’s SSD Framework Shatters Serial Bottleneck, Outperforms vLLM and SGLang
AI Explorer
AI Explorer
Mar 3, 2026 · Artificial Intelligence

How LMCache’s Lightning‑Fast KV Cache Slashes LLM First‑Token Latency

LMCache separates the KV cache from a vLLM instance into a shared service, dramatically cutting first‑token latency for repeated text, enabling multiple GPU instances to reuse cached vectors, improving hardware utilization, and supporting use cases such as long‑document QA, multi‑GPU load balancing, and prompt‑engineering, with a quick Docker‑based demo.

DockerKV CacheLLM inference
0 likes · 6 min read
How LMCache’s Lightning‑Fast KV Cache Slashes LLM First‑Token Latency
DeepHub IMBA
DeepHub IMBA
Mar 3, 2026 · Artificial Intelligence

The Evolution of KV Cache Management: From Continuous Allocation to Unified Hybrid Memory Architecture

The article traces five eras of KV cache management for LLM inference—from its absence before Transformers to the emerging unified hybrid memory architecture—comparing vLLM, SGLang, and TensorRT‑LLM and offering a decision framework for selecting the right solution in various deployment scenarios.

KV CacheLLM inferencePagedAttention
0 likes · 16 min read
The Evolution of KV Cache Management: From Continuous Allocation to Unified Hybrid Memory Architecture
MaGe Linux Operations
MaGe Linux Operations
Feb 27, 2026 · Artificial Intelligence

How to Deploy Scalable LLM Inference with vLLM on Kubernetes and GPU Scheduling

This guide explains how to deploy vLLM for large‑language‑model serving on Kubernetes, covering GPU resource management, tensor‑parallel configuration, continuous batching, quantization choices, autoscaling with HPA and KEDA, multi‑model routing, and best‑practice recommendations for performance, cost control, and high availability.

GPUKubernetesLLM inference
0 likes · 48 min read
How to Deploy Scalable LLM Inference with vLLM on Kubernetes and GPU Scheduling
PaperAgent
PaperAgent
Feb 27, 2026 · Artificial Intelligence

How DualPath Eliminates Storage Bandwidth Bottlenecks in Agentic LLM Inference

This article analyzes the DualPath architecture that redesigns KV‑Cache data paths to overcome storage‑NIC saturation in Prefill‑Decode LLM systems, presenting theoretical proofs, detailed engineering solutions, and extensive offline and online benchmarks that demonstrate up to 2.25× performance gains.

DualPathLLM inferencePerformance Optimization
0 likes · 9 min read
How DualPath Eliminates Storage Bandwidth Bottlenecks in Agentic LLM Inference
AI Engineering
AI Engineering
Jan 23, 2026 · Industry Insights

vLLM Core Team Launches Inferact, Secures $150M Seed Funding

The vLLM core maintainers have founded Inferact, raised a $150 million seed round led by Andreessen Horowitz and Lightspeed, and highlighted escalating inference challenges, the project's ecosystem dominance, and a continued commitment to open‑source development.

AI infrastructureInferactLLM inference
0 likes · 3 min read
vLLM Core Team Launches Inferact, Secures $150M Seed Funding
Alibaba Cloud Developer
Alibaba Cloud Developer
Jan 15, 2026 · Artificial Intelligence

How Hierarchical Sparse Attention Breaks KVCache Limits for Ultra‑Long Context LLMs

This article explains how a hierarchical sparse‑attention framework redesigns KVCache storage across GPU, CPU, and remote memory, eliminates bandwidth and capacity bottlenecks, and enables efficient inference for 128K‑token and larger contexts with dramatically reduced GPU memory usage and higher throughput.

Dynamic Sparse AttentionGPU memory optimizationHierarchical Storage
0 likes · 20 min read
How Hierarchical Sparse Attention Breaks KVCache Limits for Ultra‑Long Context LLMs
AI Frontier Lectures
AI Frontier Lectures
Jan 12, 2026 · Industry Insights

Why LLM Inference Hits a Memory Wall – Four Hardware Research Directions

The article analyses the challenges of large‑language‑model inference, highlighting memory bandwidth and interconnect as the primary bottlenecks, and presents four research opportunities—high‑bandwidth flash, processing‑near‑memory, 3D memory‑logic stacking, and low‑latency interconnect—while evaluating current Nvidia solutions and proposing integrated architectural approaches.

3D stackingAI hardware researchLLM inference
0 likes · 22 min read
Why LLM Inference Hits a Memory Wall – Four Hardware Research Directions
MaGe Linux Operations
MaGe Linux Operations
Jan 6, 2026 · Artificial Intelligence

How SGLang Boosted LLM Inference on H800 GPUs to 420 Tokens/s

This guide details how switching from vLLM to SGLang on eight NVIDIA H800 GPUs increased Llama‑3‑70B‑Instruct throughput from 180 to 420 tokens per second, covering SGLang’s core innovations, environment setup, configuration tweaks, performance benchmarks, troubleshooting tips, and production‑grade deployment scripts.

FlashInferGPU optimizationH800
0 likes · 19 min read
How SGLang Boosted LLM Inference on H800 GPUs to 420 Tokens/s
Alibaba Cloud Developer
Alibaba Cloud Developer
Jan 6, 2026 · Artificial Intelligence

How Tair‑KVCache‑HiSim Simulates LLM Inference 390 000× Faster with <5% Error

This article explains the design, challenges, and high‑fidelity architecture of Tair‑KVCache‑HiSim, a simulation tool that models multi‑level KV‑Cache behavior for large‑language‑model inference, predicts latency, throughput and cost under SLO constraints, and validates its predictions against real GPU deployments with sub‑5% error.

AI infrastructureCost OptimizationKVCache
0 likes · 32 min read
How Tair‑KVCache‑HiSim Simulates LLM Inference 390 000× Faster with <5% Error
Ops Community
Ops Community
Dec 28, 2025 · Artificial Intelligence

Boost LLM Inference Speed: Build a High‑Concurrency vLLM Service with Best‑Practice Ops

This guide walks through the complete process of deploying a high‑throughput large language model inference service using vLLM, covering environment preparation, installation, configuration tuning, performance testing, real‑world case studies, monitoring, troubleshooting, and backup strategies for production‑grade deployments.

GPU optimizationHigh ConcurrencyLLM inference
0 likes · 44 min read
Boost LLM Inference Speed: Build a High‑Concurrency vLLM Service with Best‑Practice Ops
Alibaba Cloud Developer
Alibaba Cloud Developer
Dec 24, 2025 · Artificial Intelligence

Boosting LLM Inference: RoleBasedGroup & Mooncake for Stable, High‑Performance Service

Large language model inference faces memory pressure, but by externalizing KVCache with Mooncake and orchestrating roles via the Kubernetes‑native RoleBasedGroup (RBG), developers can achieve stable, high‑throughput, cost‑effective serving with seamless in‑place upgrades and topology‑aware performance.

AI infrastructureKVCacheKubernetes
0 likes · 21 min read
Boosting LLM Inference: RoleBasedGroup & Mooncake for Stable, High‑Performance Service
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Dec 21, 2025 · Artificial Intelligence

Why KV Caching Is Critical for Efficient LLM Inference

The article breaks down the principles of KV caching in large language models, explaining how Q/K/V behavior differs between training and inference, the role of prompts, cache size trade‑offs, and the complexities of concurrent inference, all backed by concrete examples and references.

Concurrent InferenceKV CacheLLM inference
0 likes · 7 min read
Why KV Caching Is Critical for Efficient LLM Inference
Baidu Geek Talk
Baidu Geek Talk
Dec 10, 2025 · Artificial Intelligence

How Offloading Latent Cache Boosts DeepSeek‑V3.2‑Exp Decoding Throughput

This report analyzes the memory bottleneck of DeepSeek‑V3.2‑Exp’s sparse‑attention decoder, proposes the Expanded Sparse Server (ESS) to offload the latent cache to CPU memory, and demonstrates through high‑fidelity simulation that the approach dramatically improves decode throughput while keeping latency within acceptable limits.

Cache offloadGPU memoryLLM inference
0 likes · 20 min read
How Offloading Latent Cache Boosts DeepSeek‑V3.2‑Exp Decoding Throughput
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Dec 4, 2025 · Artificial Intelligence

How Offloading Latent Cache to CPU Boosts DeepSeek‑V3.2‑Exp Decoding Throughput

This report details the analysis of memory bottlenecks in DeepSeek‑V3.2‑Exp, proposes the Expanded Sparse Server (ESS) that offloads latent cache to CPU memory, and demonstrates through high‑fidelity simulation that the approach, combined with cache‑warmup and overlap techniques, can double decoding throughput for long‑context inference.

Cache offloadGPU‑CPU optimizationLLM inference
0 likes · 21 min read
How Offloading Latent Cache to CPU Boosts DeepSeek‑V3.2‑Exp Decoding Throughput
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Nov 25, 2025 · Artificial Intelligence

Why DeepSeek‑V3.2‑Exp Lost Performance and How a Simple RoPE Fix Restored It

The Baidu Baige team discovered that DeepSeek‑V3.2‑Exp’s long‑context performance lagged behind the official report, traced the issue to a subtle RoPE layout mismatch in the open‑source inference demo, collaborated with DeepSeek to fix it, and verified that the model’s speed and accuracy fully recovered across multiple benchmarks.

AI infrastructureDeepSeekLLM inference
0 likes · 9 min read
Why DeepSeek‑V3.2‑Exp Lost Performance and How a Simple RoPE Fix Restored It
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Nov 19, 2025 · Artificial Intelligence

Boost LLM Inference Speed with Token‑Level Two‑Chunk Overlap

Token‑level Two‑Chunk Overlap replaces traditional batch‑level Two‑Batch Overlap, dynamically splitting sequences into balanced token chunks, enabling near‑equal compute and communication times, improving GPU utilization and achieving up to 30% throughput gains in heterogeneous request workloads, with zero accuracy loss.

GPU utilizationLLM inferencePerformance Optimization
0 likes · 9 min read
Boost LLM Inference Speed with Token‑Level Two‑Chunk Overlap
Efficient Ops
Efficient Ops
Oct 14, 2025 · Artificial Intelligence

Unlock High‑Throughput LLM Inference with vLLM: Install, Run, and Optimize

This guide explains what vLLM is, how its PagedAttention architecture boosts LLM throughput, provides step‑by‑step installation commands, showcases core examples for text generation, chat, embedding and classification, and details advanced performance features such as quantization, LoRA support, and distributed parallelism.

GPU accelerationLLM inferencePython
0 likes · 8 min read
Unlock High‑Throughput LLM Inference with vLLM: Install, Run, and Optimize
Architects' Tech Alliance
Architects' Tech Alliance
Sep 30, 2025 · Artificial Intelligence

How KV Cache and CachedAttention Revolutionize LLM Inference Efficiency

This article explains how key‑value (KV) caching and the new CachedAttention technique dramatically reduce large‑language‑model inference costs by reusing stored attention data across dialogue turns, leveraging a three‑tier memory hierarchy of HBM, DRAM, and SSD to overcome bandwidth and capacity bottlenecks.

AI performanceCachedAttentionKV Cache
0 likes · 8 min read
How KV Cache and CachedAttention Revolutionize LLM Inference Efficiency
Volcano Engine Developer Services
Volcano Engine Developer Services
Jul 17, 2025 · Artificial Intelligence

How Distributed KVCache (EIC) Revolutionizes Large‑Model Inference Performance

This article examines how Volcano Engine's Elastic Instant Cache (EIC) tackles the memory bottleneck, high‑concurrency latency, and cross‑node coordination challenges of large language model inference by decoupling storage and computation, pooling resources, and applying layered optimizations, ultimately boosting AI inference efficiency, scalability, and cost‑effectiveness across various deployment scenarios.

AI infrastructureKVCacheLLM inference
0 likes · 30 min read
How Distributed KVCache (EIC) Revolutionizes Large‑Model Inference Performance
AI Algorithm Path
AI Algorithm Path
Jul 13, 2025 · Artificial Intelligence

How to Calculate the Right AI Model Size for Your PC (3B, 7B, 13B)

This article explains how to estimate the GPU memory required for running large language models of 3 B, 7 B, and 13 B parameters, walks through step‑by‑step calculations, shows how hardware limits affect feasibility, and offers practical optimization techniques such as quantization and CPU offloading.

AI model sizingCPU OffloadingFP16
0 likes · 5 min read
How to Calculate the Right AI Model Size for Your PC (3B, 7B, 13B)
Alibaba Cloud Infrastructure
Alibaba Cloud Infrastructure
May 1, 2025 · Artificial Intelligence

Fine-grained Profiling of Online AI Workloads on Kubernetes Using ACK AI Profiling

This article demonstrates how to use ACK AI Profiling, built on eBPF and dynamic process injection, to perform non-intrusive, low‑overhead profiling of Kubernetes‑deployed large‑language‑model inference services, identify GPU memory growth causes, and apply optimization recommendations to prevent OOM issues.

AI profilingGPU memoryKubernetes
0 likes · 10 min read
Fine-grained Profiling of Online AI Workloads on Kubernetes Using ACK AI Profiling
Ops Development & AI Practice
Ops Development & AI Practice
Apr 2, 2025 · Artificial Intelligence

How Cache‑Augmented Generation (CAG) Supercharges LLM Inference

Cache‑Augmented Generation (CAG) speeds up large language model text generation by caching the Transformer attention layer’s key‑value states, dramatically reducing the quadratic compute cost of autoregressive decoding while keeping the model’s knowledge unchanged.

AI performanceCAGCache‑augmented generation
0 likes · 9 min read
How Cache‑Augmented Generation (CAG) Supercharges LLM Inference
Meituan Technology Team
Meituan Technology Team
Mar 6, 2025 · Artificial Intelligence

INT8 Quantization and Inference Optimization of DeepSeek R1 Model

Meituan’s search and recommendation team converted the FP8‑only DeepSeek‑R1 model to INT8 by first casting weights to BF16 and then applying block‑wise or channel‑wise quantization, which preserves GSM8K and MMLU accuracy while delivering 33% to 50% higher throughput on A100‑80G GPUs, and they released the SGLang‑based inference scripts and quantized weights publicly, enabling deployment on older NVIDIA hardware without accuracy loss.

DeepSeek-R1GPU deploymentINT8 quantization
0 likes · 11 min read
INT8 Quantization and Inference Optimization of DeepSeek R1 Model
Architect
Architect
Mar 1, 2025 · Artificial Intelligence

How to Build a High‑Performance, Scalable LLM Inference Engine: From Paged Attention to Multi‑GPU Parallelism

This article analyzes the challenges of deploying large language models locally and presents a comprehensive set of engineering techniques—including CPU/GPU process separation, Paged Attention, Radix Attention, chunked prefill, output‑length reduction, multi‑GPU tensor parallelism, and speculative decoding—to dramatically boost inference throughput and cut response latency.

LLM inferencePerformance Optimizationchunked prefill
0 likes · 23 min read
How to Build a High‑Performance, Scalable LLM Inference Engine: From Paged Attention to Multi‑GPU Parallelism
AI Algorithm Path
AI Algorithm Path
Feb 24, 2025 · Artificial Intelligence

Flash-MLA: Boosting LLM Inference Speed on Nvidia Hopper GPUs

Flash-MLA is an open‑source GPU kernel optimized for Nvidia Hopper GPUs that compresses the KV cache of multi‑head attention, cutting memory usage by up to 93.3% and delivering 580 TFLOPS compute, thereby dramatically accelerating large‑language‑model inference while lowering cost.

DeepSeekFlash-MLAGPU optimization
0 likes · 8 min read
Flash-MLA: Boosting LLM Inference Speed on Nvidia Hopper GPUs
DaTaobao Tech
DaTaobao Tech
Oct 16, 2024 · Artificial Intelligence

Dynamic Quantization and Matrix Multiplication Optimization in MNN CPU Backend

The article details MNN’s CPU backend dynamic quantization for Transformer‑type models, describing runtime int8 conversion, block‑wise matrix‑multiply optimizations using ARM SMMLA/SDOT and AVX‑512 VNNI, weight‑group and batch‑wise quantization techniques, and reports up to three‑fold speed‑ups on Snapdragon 8 Gen 3.

CPU OptimizationDynamic QuantizationINT8
0 likes · 19 min read
Dynamic Quantization and Matrix Multiplication Optimization in MNN CPU Backend
Baobao Algorithm Notes
Baobao Algorithm Notes
Sep 29, 2024 · Artificial Intelligence

Decoding OpenAI o1: Test‑Time Scaling, PRM Search & Inference Strategies

This article analyses the training tricks behind OpenAI's o1 model, explaining test/inference‑time scaling laws, post‑training techniques, process‑supervised reward models (PRM), various inference‑time search methods, data‑collection pipelines, and the trade‑offs between allocating compute to pre‑training versus inference.

LLM inferenceOpenAI o1Reward Model
0 likes · 34 min read
Decoding OpenAI o1: Test‑Time Scaling, PRM Search & Inference Strategies
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Sep 16, 2024 · Artificial Intelligence

How TAG Makes LLM Inference Fully Asynchronous for Higher Throughput

With the growing complexity of LLM architectures like GQA, MLA, and MoE, runtime overhead has become a bottleneck; this article analyzes Python performance, communication costs, and synchronous execution in current inference frameworks, introduces the fully asynchronous TAG architecture, and demonstrates its superior throughput and latency through benchmarks.

GPU utilizationLLM inferenceRuntime Optimization
0 likes · 12 min read
How TAG Makes LLM Inference Fully Asynchronous for Higher Throughput
Alibaba Cloud Infrastructure
Alibaba Cloud Infrastructure
Sep 5, 2024 · Artificial Intelligence

Deploying NVIDIA NIM on Alibaba Cloud ACK with Cloud‑Native AI Suite: A Step‑by‑Step Guide

This guide explains how to quickly build a high‑performance, observable, and elastically scalable LLM inference service by deploying NVIDIA NIM on an Alibaba Cloud ACK cluster using the Cloud‑Native AI Suite, KServe, Prometheus, Grafana, and custom autoscaling based on request‑queue metrics.

Alibaba Cloud ACKAutoscalingGrafana
0 likes · 15 min read
Deploying NVIDIA NIM on Alibaba Cloud ACK with Cloud‑Native AI Suite: A Step‑by‑Step Guide
Baobao Algorithm Notes
Baobao Algorithm Notes
Jul 11, 2024 · Artificial Intelligence

Why Separate Prefill and Decode? A Deep Dive into DistServe’s Split Inference Architecture

This article explores the two‑stage LLM inference pipeline, introduces TTFT and TPOT metrics, explains the motivation for prefilling‑decoding separation, presents experimental comparisons between split and merged architectures, and details optimization techniques and parallel‑strategy modeling for DistServe.

DistServeGoodputLLM inference
0 likes · 28 min read
Why Separate Prefill and Decode? A Deep Dive into DistServe’s Split Inference Architecture
Architect
Architect
Jul 2, 2024 · Artificial Intelligence

Mooncake: A Separated Architecture for Large‑Language‑Model Inference

The article presents Mooncake, a split‑architecture inference platform for the Kimi LLM assistant, detailing its three elastic resource pools, the rationale for using Time‑Between‑Tokens over TPOT, and design choices for Prefill, KVCache, and Decode stages to improve latency and throughput.

AI SystemsDistributed ArchitectureKVCache
0 likes · 9 min read
Mooncake: A Separated Architecture for Large‑Language‑Model Inference