BestHub
Sign in
Tagged articles

vLLM

148 articles · Page 1 of 2
Machine Heart
Machine Heart
Jul 5, 2026 · Artificial Intelligence

Is the Router’s Role Underrated? How vLLM Turns a Single Call into a Model Collaboration Squad

The article analyzes how routers have evolved from simple request forwarders into intelligent orchestrators that manage cost, safety, and cloud‑edge collaboration, detailing vLLM’s Semantic Router, its micro‑agent loop patterns, experimental benchmarks, and the resulting hybrid model serving architecture.

AI RoutingInference ServingMicro-Agents
0 likes · 13 min read
Is the Router’s Role Underrated? How vLLM Turns a Single Call into a Model Collaboration Squad
Raymond Ops
Raymond Ops
Jul 2, 2026 · Operations

How to Monitor Large Model Applications: A Beginner‑Friendly Metric System

This guide walks you through building a production‑grade monitoring solution for large language model inference services using a three‑layer metric hierarchy, Prometheus, Grafana, DCGM Exporter, and custom Python metrics, with step‑by‑step deployment, alerting policies, and real‑world troubleshooting examples.

AI InfrastructureLarge Language ModelsMonitoring
0 likes · 42 min read
How to Monitor Large Model Applications: A Beginner‑Friendly Metric System
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 29, 2026 · Artificial Intelligence

How Nvidia’s NVFP4 Cuts GLM‑5.2 Deployment Cost by Half

Semgrep’s benchmark shows open‑source GLM‑5.2 matching Claude’s performance while costing only $0.17 per vulnerability, and Nvidia’s NVFP4 quantization halves the model’s memory footprint with virtually unchanged accuracy, making local deployment on 8‑GPU systems far more affordable.

AI DeploymentGLM-5.2Model Quantization
0 likes · 11 min read
How Nvidia’s NVFP4 Cuts GLM‑5.2 Deployment Cost by Half
Raymond Ops
Raymond Ops
Jun 27, 2026 · Artificial Intelligence

vLLM Quantized Inference: Loading AWQ/GPTQ Models and Optimizing GPU Memory

This article provides a step‑by‑step guide on using vLLM to load AWQ and GPTQ quantized large language models, covering environment setup, calibration data preparation, model quantization, deployment scripts, performance benchmarking, accuracy checks, best‑practice recommendations, and troubleshooting tips for GPU memory optimization.

AWQGPTQGPU memory optimization
0 likes · 45 min read
vLLM Quantized Inference: Loading AWQ/GPTQ Models and Optimizing GPU Memory
Raymond Ops
Raymond Ops
Jun 22, 2026 · Artificial Intelligence

Elastic Deployment and GPU Scheduling for Large‑Model Inference with vLLM on Kubernetes

This article presents a detailed, step‑by‑step analysis of deploying the high‑performance vLLM inference engine on Kubernetes, covering GPU memory management, tensor parallelism, quantization choices, continuous batching, and automated scaling with HPA/KEDA to achieve low latency and high throughput for large language models.

DockerGPU schedulingKubernetes
0 likes · 49 min read
Elastic Deployment and GPU Scheduling for Large‑Model Inference with vLLM on Kubernetes
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 15, 2026 · Artificial Intelligence

vLLM 0.23.0 Brings Faster Local LLM Deployment and Wider Hardware Support

Version 0.23.0 of the open‑source vLLM inference engine adds full DeepSeek‑V4 stability, Model Runner V2 coverage for Llama, Mistral, Qwen3 and new models, a production‑grade Rust front‑end, multi‑level KV‑cache offloading, extensive hardware optimizations across NVIDIA, AMD, Intel, TPU and RISC‑V, plus API enhancements, delivering up to 20 % performance gains while simplifying deployment.

DeepSeek-V4KV cache offloadingLLM Inference
0 likes · 8 min read
vLLM 0.23.0 Brings Faster Local LLM Deployment and Wider Hardware Support
Geek Labs
Geek Labs
Jun 12, 2026 · Artificial Intelligence

Boost Developer Productivity with NBD VRAM, Reg Factory, and vLLM Studio

This article introduces three open‑source tools that improve GPU‑centric development: NBD VRAM, which turns GPU memory into Linux swap space; Reg Factory, a scheduler and monitor for multi‑GPU clusters; and vLLM Studio, a web UI for deploying and managing large‑model inference.

GPUModel DeploymentOpen-source
0 likes · 4 min read
Boost Developer Productivity with NBD VRAM, Reg Factory, and vLLM Studio
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 11, 2026 · Artificial Intelligence

Google’s 26B DiffusionGemma Model Delivers 1000+ Tokens/s – Runs on a 4090

DiffusionGemma, Google DeepMind’s 26B MoE model that generates 256‑token blocks via diffusion, achieves over 1000 tokens per second on H100/H200 GPUs, offers FP8 and NVFP4 quantized versions with near‑lossless accuracy, and can be deployed locally with vLLM Docker images, though it incurs higher first‑token latency and limited concurrency.

26B modelDiffusionGemmaFP8 quantization
0 likes · 10 min read
Google’s 26B DiffusionGemma Model Delivers 1000+ Tokens/s – Runs on a 4090
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 7, 2026 · Artificial Intelligence

Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained

The article explains why LLM inference is memory‑bound, introduces vLLM’s three core optimizations—Continuous Batching, PagedAttention, and Prefix Caching—shows how to launch a vLLM server, run Python code to benchmark performance, and examines KV‑Cache memory usage with concrete numbers.

Continuous BatchingKV cacheLLM Inference
0 likes · 11 min read
Hands‑On LLM Local Deployment: vLLM Inference Optimizations Explained
Old Zhang's AI Learning
Old Zhang's AI Learning
May 31, 2026 · Artificial Intelligence

Qwen3.6-35B-A3B NVFP4: A Stable, Highly Compressed Quantized Model

NVIDIA's NVFP4 quantization reduces Qwen3.6-35B-A3B's memory footprint by threefold with almost no accuracy loss, offers plug‑and‑play deployment via vLLM, and outperforms other 4‑bit formats on Hopper/Blackwell GPUs, making it a practical choice for production AI workloads.

BenchmarkMoENVFP4
0 likes · 13 min read
Qwen3.6-35B-A3B NVFP4: A Stable, Highly Compressed Quantized Model
Old Zhang's AI Learning
Old Zhang's AI Learning
May 31, 2026 · Artificial Intelligence

vLLM 0.22 Release: Production-Ready DeepSeek V4 and Extreme KV Cache Compression

The vLLM 0.22 stable release introduces production‑grade DeepSeek V4 support, massive kernel fusions, up to 10‑20× speedups, Batch Invariance with 28.9% latency gain, a Rust front‑end, multi‑level KV cache offload that can double context length, and broad hardware coverage across NVIDIA, AMD, CPU and RISC‑V, making it a pivotal upgrade for inference infrastructure teams.

Batch InvarianceDeepSeek-V4Inference Optimization
0 likes · 13 min read
vLLM 0.22 Release: Production-Ready DeepSeek V4 and Extreme KV Cache Compression
Old Zhang's AI Learning
Old Zhang's AI Learning
May 30, 2026 · Artificial Intelligence

vLLM Introduces Native RL API for Seamless Weight Synchronization

vLLM’s new native RL API introduces a four‑stage weight‑transfer protocol, pluggable backends, and a keep‑mode pause/resume mechanism that eliminates deadlocks in DPEP deployments, with large‑scale validations on SkyRL and Prime‑RL demonstrating reliability and performance gains.

CUDA IPCDistributed InferenceNCCL
0 likes · 14 min read
vLLM Introduces Native RL API for Seamless Weight Synchronization
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 25, 2026 · Artificial Intelligence

VeRL-Omni: A Universal RL Post‑Training Framework for Diffusion and Multimodal Generation Models

VeRL-Omni introduces a universal reinforcement‑learning post‑training framework that extends the verl and vLLM‑Omni stacks to support diffusion transformers, hybrid AR‑DiT, and unified understanding‑generation models, offering high‑throughput multimodal rollout, flexible reward engines, modular trainers, and broad hardware compatibility.

FlowGRPOMultimodal GenerationRL
0 likes · 9 min read
VeRL-Omni: A Universal RL Post‑Training Framework for Diffusion and Multimodal Generation Models
Tencent Technical Engineering
Tencent Technical Engineering
May 25, 2026 · Artificial Intelligence

vLLM Deep Dive: Continuous Batching and Paged Attention for Fast LLM Inference

This article walks through a two‑month source‑code study of vLLM, explaining how token‑level scheduling, continuous batching, and the Paged Attention mechanism reshape tensor dimensions to turn large‑model inference into a compute‑bound, high‑throughput process while managing GPU memory efficiently.

Continuous BatchingFlashAttentionGPU Optimization
0 likes · 29 min read
vLLM Deep Dive: Continuous Batching and Paged Attention for Fast LLM Inference
Machine Heart
Machine Heart
May 25, 2026 · Artificial Intelligence

VeRL-Omni: Universal RL Post‑Training for Diffusion and Multimodal Models

VeRL-Omni is an open‑source RL post‑training framework built on verl and vLLM‑Omni that enables efficient, high‑throughput rollout and flexible reward computation for diffusion, AR‑DiT, and unified multimodal generation models, supporting diverse hardware, modular trainers, and demonstrating up to 14% latency reduction and high training throughput in benchmark experiments.

Diffusion ModelsFlowGRPOMultimodal Generation
0 likes · 9 min read
VeRL-Omni: Universal RL Post‑Training for Diffusion and Multimodal Models
Old Zhang's AI Learning
Old Zhang's AI Learning
May 24, 2026 · Industry Insights

How a Fake vLLM PR Exposed the Risks of AI‑Generated Resume Padding

The article dissects a fabricated vLLM pull request that pretended to fix a non‑existent NVIDIA Eagle3 checkpoint bug, explains its bogus test plan, shows how AI‑assisted PR generation can flood open‑source projects, and warns of the trust damage such resume‑padding schemes cause.

AI coding agentsEagle3NVIDIA
0 likes · 7 min read
How a Fake vLLM PR Exposed the Risks of AI‑Generated Resume Padding
Old Zhang's AI Learning
Old Zhang's AI Learning
May 16, 2026 · Artificial Intelligence

vLLM 0.21.0 Arrives: Speculative Decoding Now Supports Reasoning Models

The vLLM 0.21.0 release brings five major updates—including Transformers v4 deprecation, a C++20 build requirement, KV offload with hybrid memory, speculative decoding that respects thinking budgets, and a Blackwell token‑speed backend—while offering detailed upgrade guidance for different user groups.

C++20KV cacheLarge Language Models
0 likes · 12 min read
vLLM 0.21.0 Arrives: Speculative Decoding Now Supports Reasoning Models
Old Zhang's AI Learning
Old Zhang's AI Learning
May 13, 2026 · Artificial Intelligence

Why vLLM Now Leads Open‑Source LLM Inference Benchmarks

vLLM tops the Artificial Analysis ranking by delivering the highest throughput for DeepSeek V3.2, Qwen 3.5 397B, and MiniMax‑M2.5 on identical NVIDIA Blackwell Ultra hardware, thanks to extensive kernel‑fusion optimizations that remain in the main branch.

DeepSeekLLM InferenceQwen
0 likes · 7 min read
Why vLLM Now Leads Open‑Source LLM Inference Benchmarks
Lao Guo's Learning Space
Lao Guo's Learning Space
May 12, 2026 · Artificial Intelligence

Which Inference Framework Maximizes Your GPU Performance in 2026?

This article compares six popular LLM inference frameworks—vLLM, TensorRT‑LLM, llama.cpp, ds4.c, Ollama, and Omlx—across performance, ease of use, and hardware compatibility, then provides a practical matrix to help users select the best fit for their GPU.

Apple SiliconGPU performanceLLM Inference
0 likes · 10 min read
Which Inference Framework Maximizes Your GPU Performance in 2026?
Machine Heart
Machine Heart
May 7, 2026 · Artificial Intelligence

Nvidia Endorses TokenSpeed: A Light‑Speed Agent Inference Engine Built in Two Months

TokenSpeed, an open‑source LLM inference engine designed for agent workloads, delivers TensorRT‑LLM‑level performance and vLLM‑level ease of use, outperforms TensorRT‑LLM by up to 11% throughput and halves latency on speculative decoding, and has earned Nvidia’s public recommendation.

Agent workloadsLLM InferenceNVIDIA Blackwell
0 likes · 8 min read
Nvidia Endorses TokenSpeed: A Light‑Speed Agent Inference Engine Built in Two Months
Geek Labs
Geek Labs
May 7, 2026 · Artificial Intelligence

Running Large Language Models Locally on RTX 3090: Two Open‑Source Solutions

This article introduces two recent GitHub projects—club‑3090, which enables single‑ or dual‑RTX 3090 inference of 27‑billion‑parameter models with detailed performance benchmarks, and library‑skills, a tool that keeps AI agents synchronized with the latest official library APIs—explaining their configurations, usage steps, hardware requirements, and target audiences.

AI agentsDockerLarge Language Models
0 likes · 7 min read
Running Large Language Models Locally on RTX 3090: Two Open‑Source Solutions
Old Zhang's AI Learning
Old Zhang's AI Learning
May 6, 2026 · Artificial Intelligence

Google Boosts Gemma 4 Inference Speed Up to 3× with MTP Drafter and Day‑0 vLLM Support

Google’s new Multi‑Token Prediction (MTP) drafter for Gemma 4 delivers up to three‑fold inference speedups across hardware and frameworks—validated by official benchmarks and independent DGX Spark tests—while preserving identical output quality, and is immediately usable via Hugging Face, vLLM, MLX, Ollama and edge‑device runtimes.

Apple SiliconGemma 4LLM Inference
0 likes · 9 min read
Google Boosts Gemma 4 Inference Speed Up to 3× with MTP Drafter and Day‑0 vLLM Support
Old Zhang's AI Learning
Old Zhang's AI Learning
May 5, 2026 · Artificial Intelligence

vLLM 0.20.1 Fixes Instability and Speed Issues for DeepSeek V4

The vLLM 0.20.1 patch, released shortly after 0.20.0, consolidates stability fixes and performance optimizations for DeepSeek V4, adds several bug fixes, updates installation instructions, and provides targeted upgrade recommendations for different user scenarios.

Bug FixDeepSeek-V4GPU inference
0 likes · 9 min read
vLLM 0.20.1 Fixes Instability and Speed Issues for DeepSeek V4
Linyb Geek Road
Linyb Geek Road
May 5, 2026 · Artificial Intelligence

Optimizing Retrieval and Generation Latency in High‑Concurrency RAG Agents

The article dissects latency in high‑concurrency RAG Agent pipelines, showing how retrieval, re‑ranking, and LLM generation each contribute milliseconds of delay, and presents system‑level tactics—from ANN index tuning and partitioned search to vLLM PagedAttention, continuous batching, speculative decoding, model quantization, routing, semantic caching, and pipeline parallelism—to dramatically cut end‑to‑end response time.

ANNLLMRAG
0 likes · 15 min read
Optimizing Retrieval and Generation Latency in High‑Concurrency RAG Agents
Old Zhang's AI Learning
Old Zhang's AI Learning
May 1, 2026 · Artificial Intelligence

NVIDIA’s Open‑Source Multimodal Nemotron 3 Nano Omni: Run Locally on Consumer GPUs (English‑Only)

NVIDIA’s Nemotron 3 Nano Omni 30B‑A3B‑Reasoning model, an open‑source multimodal LLM with 30 B parameters, 256K context and video‑audio‑image‑text capabilities, outperforms comparable models by up to 9.2× in video throughput, runs on consumer GPUs via 4‑bit GGUF quantization, but currently supports only English input.

GGUFGPUMultimodal
0 likes · 17 min read
NVIDIA’s Open‑Source Multimodal Nemotron 3 Nano Omni: Run Locally on Consumer GPUs (English‑Only)
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 28, 2026 · Artificial Intelligence

vLLM 0.20 Arrives with DeepSeek V4 Support – What’s New?

The vLLM 0.20.0 release dramatically upgrades the inference engine with DeepSeek V4 support, default CUDA 13, PyTorch 2.11, Transformers v5 compatibility, FlashAttention 4 MLA prefill, TurboQuant 2‑bit KV cache, an online quantization front‑end, IR enhancements, Model Runner V2 features, and a slew of new models, while providing detailed installation and upgrade guidance.

CUDA 13DeepSeek-V4FlashAttention
0 likes · 10 min read
vLLM 0.20 Arrives with DeepSeek V4 Support – What’s New?
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 26, 2026 · Artificial Intelligence

Why Deploying DeepSeek‑V4 Locally with vLLM Is So Challenging

The article dissects DeepSeek‑V4’s local deployment using vLLM, explaining the steep hardware requirements, the complex heterogeneous KV‑cache architecture, and the aggressive kernel‑fusion and multi‑stream optimizations that together make high‑context inference both memory‑intensive and engineering‑heavy.

DeepSeek-V4GPU memoryKV cache
0 likes · 15 min read
Why Deploying DeepSeek‑V4 Locally with vLLM Is So Challenging
Woodpecker Software Testing
Woodpecker Software Testing
Apr 24, 2026 · Artificial Intelligence

Practical Guide to Optimizing Large Model Performance in Production

This guide details how enterprises can move large language models from lab to production by defining specific SLI/SLO metrics, diagnosing hidden bottlenecks such as tokenizer latency, and applying four quantifiable optimization levers that dramatically improve latency, throughput, and cost efficiency.

Continuous BatchingGPU OptimizationLarge Language Models
0 likes · 6 min read
Practical Guide to Optimizing Large Model Performance in Production
Machine Heart
Machine Heart
Apr 24, 2026 · Artificial Intelligence

Cambricon Achieves Day‑0 Native Support for DeepSeek‑V4, Uniting Two Chinese AI Leaders

Cambricon leveraged its NeuWare stack and vLLM framework to deliver Day‑0 native support for DeepSeek‑V4‑flash (285 B) and DeepSeek‑V4‑pro (1.6 T), open‑sourcing the adaptation and showcasing rapid model migration alongside extreme performance optimizations across software and hardware layers.

AI inferenceCambriconDeepSeek-V4
0 likes · 5 min read
Cambricon Achieves Day‑0 Native Support for DeepSeek‑V4, Uniting Two Chinese AI Leaders
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 20, 2026 · Artificial Intelligence

Kimi K2.6: The Most Powerful Open-Source Agent Model – Architecture, Benchmarks, and Deployment Guide

Kimi K2.6, an open-source 1-trillion-parameter MoE model, expands Agent capabilities with 256K context, multimodal inputs, and the ability to coordinate 300 sub-Agents over 4,000 steps, achieving top scores on benchmarks like Terminal-Bench 2.0, SWE-Bench Pro, and BrowseComp, while offering flexible deployment via vLLM, SGLang, and KTransformers.

Agent ModelBenchmarkDeployment
0 likes · 11 min read
Kimi K2.6: The Most Powerful Open-Source Agent Model – Architecture, Benchmarks, and Deployment Guide
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 19, 2026 · Artificial Intelligence

Qwen3.6-35B: 4‑bit Quantization, DFlash Speedup, Claude Opus Distillation

The article reviews three optimization paths for the Qwen3.6‑35B model—four‑bit AWQ quantization variants, the DFlash speculative decoding accelerator, and a Claude Opus‑based distillation—detailing their implementation steps, benchmark results, and guidance on selecting the best version for different hardware and performance needs.

AIDFlashDistillation
0 likes · 11 min read
Qwen3.6-35B: 4‑bit Quantization, DFlash Speedup, Claude Opus Distillation
Lao Guo's Learning Space
Lao Guo's Learning Space
Apr 19, 2026 · Artificial Intelligence

Which Framework Wins for Running Large Models? vLLM vs llama.cpp vs MLX (2026 Deep Comparison)

The article provides a 2026 deep comparative analysis of three major large‑model inference frameworks—vLLM, llama.cpp, and MLX—detailing their core designs, recent updates, benchmark results on various hardware, deployment complexity, and recommended use cases to help developers choose the right tool.

BenchmarkMLXframework comparison
0 likes · 15 min read
Which Framework Wins for Running Large Models? vLLM vs llama.cpp vs MLX (2026 Deep Comparison)
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 14, 2026 · Artificial Intelligence

Qwen3.5-27B-DFlash Delivers Up to 5× Faster Inference Without Quality Loss

The DFlash approach replaces speculative decoding’s autoregressive drafter with a block diffusion model and injects target‑model hidden features into every KV‑cache layer, achieving up to 5× speed‑up for Qwen3.5‑27B on single‑GPU and 1.5–1.9× on high‑concurrency workloads while preserving output quality.

DFlashQwen3.5SGLang
0 likes · 12 min read
Qwen3.5-27B-DFlash Delivers Up to 5× Faster Inference Without Quality Loss
SuanNi
SuanNi
Apr 13, 2026 · Artificial Intelligence

Deploy Qwen3 8B Model with vLLM: Step‑by‑Step Guide for Remote Inference

This guide walks you through deploying Alibaba’s open‑source Qwen‑3 8B model on the SumW platform using vLLM, covering environment activation, server launch with OpenAI‑compatible parameters, SSH tunneling for remote access, and Python client calls, while highlighting key configuration tips and common pitfalls.

Model DeploymentOpenAI APIPython SDK
0 likes · 6 min read
Deploy Qwen3 8B Model with vLLM: Step‑by‑Step Guide for Remote Inference
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 12, 2026 · Artificial Intelligence

Deploy the Open‑Source MiniMax‑M2.7 Model Locally: Step‑by‑Step Guide

MiniMax‑M2.7, the newly open‑sourced 230‑billion‑parameter MoE model, offers self‑evolution, professional software engineering and agent capabilities, and can be deployed locally using Ollama, vLLM, SGLang or Docker with 4‑8 H200 GPUs, while the article details hardware needs, performance gains and tool‑calling/Thinking features.

DeploymentGPULLM
0 likes · 11 min read
Deploy the Open‑Source MiniMax‑M2.7 Model Locally: Step‑by‑Step Guide
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 10, 2026 · Artificial Intelligence

How a 9B‑parameter Qwen3.5 model achieves full‑auto data analysis on a consumer GPU

The open‑source CoPaw‑Flash‑9B‑DataAnalyst‑LoRA model, fine‑tuned via LoRA, can autonomously load, explore, statistically analyze, visualize, and generate structured reports for CSV/Excel/JSON datasets, achieving a 90% success rate with an average of 26 iteration rounds, and it runs on a single consumer‑grade GPU using vLLM and the Data Analyst framework.

AgentData AnalystGPU
0 likes · 10 min read
How a 9B‑parameter Qwen3.5 model achieves full‑auto data analysis on a consumer GPU
AI Tech Publishing
AI Tech Publishing
Apr 9, 2026 · Artificial Intelligence

Engineering‑Focused Guide to Training and Inference of Large Language Models

This article walks engineers through the full LLM stack—from tokenization and positional encoding to transformer blocks, efficient fine‑tuning, quantization, and production‑grade inference techniques such as KV‑cache, FlashAttention, PagedAttention, continuous batching, and speculative decoding—highlighting trade‑offs, toolchains, and practical workflow steps.

LLMLoRAQuantization
0 likes · 13 min read
Engineering‑Focused Guide to Training and Inference of Large Language Models
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Apr 8, 2026 · Artificial Intelligence

Unlocking 8‑Hour Autonomous Coding: GLM‑5.1’s Leap with Kunlun XPU

The open‑source GLM‑5.1 model, adapted to Baidu Baige's Kunlun XPU via the vLLM‑Kunlun Plugin, delivers record‑breaking SWE‑bench scores, eight‑hour autonomous coding, long‑context handling up to 64K tokens, and scalable deployment across tens of thousands of chips, showcasing end‑to‑end AI acceleration.

GLM-5.1Kunlun XPULong Context
0 likes · 8 min read
Unlocking 8‑Hour Autonomous Coding: GLM‑5.1’s Leap with Kunlun XPU
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 7, 2026 · Artificial Intelligence

vLLM 0.19.0: HuggingFace v5 Support, Multimodal Boosts, and CPU KV Cache Offload

The vLLM 0.19.0 release adds first‑day Gemma 4 support, merges zero‑bubble asynchronous scheduling with speculative decoding, matures Model Runner V2, introduces full‑CUDA‑graph acceleration for ViT, generalizes DBO, brings CPU KV cache offload, and expands hardware and Transformers compatibility, offering substantial performance and flexibility gains for production LLM inference.

CPU KV offloadGPUGemma 4
0 likes · 18 min read
vLLM 0.19.0: HuggingFace v5 Support, Multimodal Boosts, and CPU KV Cache Offload
DeepHub IMBA
DeepHub IMBA
Apr 2, 2026 · Artificial Intelligence

Speculative Decoding Explained: Small Draft Model + One‑Shot Verification

The article details how speculative decoding—using a fast small model to draft tokens and a large model to verify them—overcomes the memory‑bandwidth bottleneck of autoregressive inference, introduces SSD’s self‑draft and tree‑verification stages, presents real‑world benchmark gains, and shows how to enable it in vLLM.

GPU memory bandwidthInference OptimizationLarge Language Models
0 likes · 14 min read
Speculative Decoding Explained: Small Draft Model + One‑Shot Verification
Old Zhang's AI Learning
Old Zhang's AI Learning
Mar 27, 2026 · Artificial Intelligence

vLLM’s Four Major 2026 Updates: Semantic Router Athena, Nemotron 3 Super, P‑EAGLE, and Model Runner V2

The March 2026 vLLM release bundle introduces four substantial upgrades—Semantic Router v0.2 Athena, NVIDIA Nemotron 3 Super, the parallel speculative decoding P‑EAGLE, and a completely re‑architected Model Runner V2—each backed by concrete benchmarks, architectural diagrams, and code examples that demonstrate how the engine evolves from a pure inference engine to a full‑stack AI serving platform.

GPU AccelerationModel Runner V2Nemotron-3-Super
0 likes · 17 min read
vLLM’s Four Major 2026 Updates: Semantic Router Athena, Nemotron 3 Super, P‑EAGLE, and Model Runner V2
Architect's Ambition
Architect's Ambition
Mar 25, 2026 · Artificial Intelligence

From Zero to Production: Building AI‑Native Infrastructure for Agents – Local Inference to Full‑Scale Deployment

The article walks through constructing AI‑native infrastructure for agents, covering local inference deployment with vLLM, setting up an AI gateway using LiteLLM, implementing observability with logs, metrics, and tracing, and applying cost‑saving strategies that reduced latency, improved stability, and cut expenses by up to 60%.

AI agentsDeploymentDocker
0 likes · 13 min read
From Zero to Production: Building AI‑Native Infrastructure for Agents – Local Inference to Full‑Scale Deployment
Weekly Large Model Application
Weekly Large Model Application
Mar 23, 2026 · Artificial Intelligence

Inside Step‑Audio2: End‑to‑End Multimodal Audio LLM Architecture and Deployment

This article dissects Step‑Audio2, an industrial‑grade multimodal large language model that unifies speech understanding, translation, dialogue and audio generation in a single causal LM, detailing its inference pipeline, key implementation tricks, deployment modes, strengths, limitations, and suitable application scenarios.

PythonSpeech synthesisStep-Audio2
0 likes · 10 min read
Inside Step‑Audio2: End‑to‑End Multimodal Audio LLM Architecture and Deployment
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Mar 23, 2026 · Artificial Intelligence

How vLLM‑Kunlun Unlocks Peak LLM Performance on Kunlun XPU

This article details the technical challenges of adapting the open‑source vLLM inference framework to Baidu's Kunlun XPU, outlines four major performance bottlenecks, and presents a multi‑dimensional optimization roadmap—including custom plugins, operator fusion, INT8 quantization, and CUDA‑Graph techniques—that together boost throughput by up to 8% and narrow the gap with leading GPU hardware.

CUDA GraphINT8 QuantizationKunlun XPU
0 likes · 13 min read
How vLLM‑Kunlun Unlocks Peak LLM Performance on Kunlun XPU
Fun with Large Models
Fun with Large Models
Mar 20, 2026 · Artificial Intelligence

Step‑by‑Step Guide to Using LLaMAFactory for Full‑Cycle Large‑Model Training (Part 9)

This article walks through the complete workflow of fine‑tuning a Qwen2.5‑0.5B model with LLaMAFactory, covering environment setup, model download, dataset preparation, configuration editing, training execution, LoRA weight merging, and deployment via vLLM, while highlighting the framework’s minimal‑code and broad model support.

AI model trainingLLaMAFactoryLoRA
0 likes · 12 min read
Step‑by‑Step Guide to Using LLaMAFactory for Full‑Cycle Large‑Model Training (Part 9)
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Mar 18, 2026 · Artificial Intelligence

How vLLM‑Kunlun Brings CUDA‑Like Inference to Kunlun XPU: Architecture, Adaptation, and Performance Wins

This article details the vLLM‑Kunlun open‑source project that adapts the high‑performance vLLM inference engine to Baidu's Kunlun XPU, covering platform overview, model‑porting workflow, plugin architecture, concrete case studies with MIMO‑Flash‑V2 and Qwen 3.5, and the performance‑tuning techniques that enable seamless, GPU‑level inference on domestic hardware.

AIHardwareKunlun
0 likes · 12 min read
How vLLM‑Kunlun Brings CUDA‑Like Inference to Kunlun XPU: Architecture, Adaptation, and Performance Wins
Ops Community
Ops Community
Mar 13, 2026 · Backend Development

How to Diagnose and Fix Slow LLM Inference: A Full‑Stack Performance Guide

This article presents a comprehensive, step‑by‑step methodology for troubleshooting and optimizing large‑language‑model inference performance, covering GPU, CPU, memory, network, configuration, and application layers, with concrete benchmark scripts, diagnostic commands, and real‑world case studies.

CPUGPUPerformance
0 likes · 48 min read
How to Diagnose and Fix Slow LLM Inference: A Full‑Stack Performance Guide
MaGe Linux Operations
MaGe Linux Operations
Mar 12, 2026 · Backend Development

How to Deploy vLLM Inference Service on Kubernetes with Ingress and Service Load Balancing

This guide walks through deploying a production‑grade vLLM inference service on Kubernetes, covering GPU resource scheduling, Service and Ingress configuration, session affinity, health checks, performance tuning, scaling, monitoring, fault‑tolerance, and best‑practice recommendations for high‑availability AI workloads.

GPUHigh AvailabilityIngress
0 likes · 47 min read
How to Deploy vLLM Inference Service on Kubernetes with Ingress and Service Load Balancing
Old Zhang's AI Learning
Old Zhang's AI Learning
Mar 7, 2026 · Artificial Intelligence

vLLM 0.17.0 Release: Full Qwen 3.5 Support and Anthropic API Compatibility

The vLLM 0.17.0 release brings FlashAttention 4 integration, a mature Model Runner V2, complete Qwen 3.5 series support, a one‑click performance‑mode flag, Anthropic API compatibility, advanced weight‑offloading, broader hardware support beyond NVIDIA, ASR model integration, and detailed upgrade and installation guidance.

ASRAnthropic APIFlashAttention
0 likes · 12 min read
vLLM 0.17.0 Release: Full Qwen 3.5 Support and Anthropic API Compatibility
AI Explorer
AI Explorer
Mar 3, 2026 · Artificial Intelligence

How LMCache’s Lightning‑Fast KV Cache Slashes LLM First‑Token Latency

LMCache separates the KV cache from a vLLM instance into a shared service, dramatically cutting first‑token latency for repeated text, enabling multiple GPU instances to reuse cached vectors, improving hardware utilization, and supporting use cases such as long‑document QA, multi‑GPU load balancing, and prompt‑engineering, with a quick Docker‑based demo.

DockerKV cacheLLM Inference
0 likes · 6 min read
How LMCache’s Lightning‑Fast KV Cache Slashes LLM First‑Token Latency
DeepHub IMBA
DeepHub IMBA
Mar 3, 2026 · Artificial Intelligence

The Evolution of KV Cache Management: From Continuous Allocation to Unified Hybrid Memory Architecture

The article traces five eras of KV cache management for LLM inference—from its absence before Transformers to the emerging unified hybrid memory architecture—comparing vLLM, SGLang, and TensorRT‑LLM and offering a decision framework for selecting the right solution in various deployment scenarios.

KV cacheLLM InferenceMemory Management
0 likes · 16 min read
The Evolution of KV Cache Management: From Continuous Allocation to Unified Hybrid Memory Architecture
MaGe Linux Operations
MaGe Linux Operations
Feb 27, 2026 · Artificial Intelligence

How to Deploy Scalable LLM Inference with vLLM on Kubernetes and GPU Scheduling

This guide explains how to deploy vLLM for large‑language‑model serving on Kubernetes, covering GPU resource management, tensor‑parallel configuration, continuous batching, quantization choices, autoscaling with HPA and KEDA, multi‑model routing, and best‑practice recommendations for performance, cost control, and high availability.

GPUKubernetesLLM Inference
0 likes · 48 min read
How to Deploy Scalable LLM Inference with vLLM on Kubernetes and GPU Scheduling
Baobao Algorithm Notes
Baobao Algorithm Notes
Feb 25, 2026 · Artificial Intelligence

Exploring Qwen 3.5: Small‑Scale MoE Models, Architecture, and Deployment Guides

This article reviews the three open‑source Qwen 3.5 models—including a 35B MoE, a 122B MoE, and a 27B dense version—detailing their parameter layouts, core attention designs, context length, inference performance, hardware requirements, and provides step‑by‑step code examples for loading them with Hugging Face Transformers and vLLM.

AILarge Language ModelMoE
0 likes · 10 min read
Exploring Qwen 3.5: Small‑Scale MoE Models, Architecture, and Deployment Guides
AI Engineering
AI Engineering
Feb 16, 2026 · Artificial Intelligence

Qwen3.5-397B: 397B‑Parameter Multimodal LLM Boosts Inference Speed 8‑19×

Alibaba’s Qwen3.5-397B-A17B, a 397‑billion‑parameter open‑source multimodal LLM, combines mixed linear attention with a sparse MoE architecture to achieve 8.6‑19× higher decoding throughput than Qwen3‑Max, supports 201 languages, and can be deployed via vLLM, Docker, Transformers, or SGLang with various optimization presets.

Inference OptimizationLarge Language ModelQwen3.5
0 likes · 8 min read
Qwen3.5-397B: 397B‑Parameter Multimodal LLM Boosts Inference Speed 8‑19×
Node.js Tech Stack
Node.js Tech Stack
Feb 16, 2026 · Artificial Intelligence

Qwen 3.5 Launch: 17B Active Parameters Take on GPT‑5.2

Qwen 3.5, an open‑source 397B‑parameter model that activates only 17B parameters, uses a hybrid MoE‑Gated Delta architecture, offers native multimodal support and a default chain‑of‑thought mode, and achieves benchmark scores comparable to GPT‑5.2, Claude 4.5 Opus and Gemini 3 Pro across code, math, agent and vision tasks.

AI modelBenchmarkGated Delta Networks
0 likes · 9 min read
Qwen 3.5 Launch: 17B Active Parameters Take on GPT‑5.2
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Feb 12, 2026 · Artificial Intelligence

Deploying GLM-5 on Baidu Kunlun P800 XPU with vLLM‑Kunlun Plugin

This article explains how Baidu's new GLM-5 large model is adapted to the Kunlun P800 XPU, detailing the async reinforcement learning framework Slime, optimization techniques like INT8 quantization and tensor‑parallelism, and provides step‑by‑step deployment commands using the open‑source vLLM‑Kunlun plugin.

AI accelerationGLM-5INT8 Quantization
0 likes · 6 min read
Deploying GLM-5 on Baidu Kunlun P800 XPU with vLLM‑Kunlun Plugin
AI Engineering
AI Engineering
Feb 12, 2026 · Artificial Intelligence

GLM-5 Unveiled: 744B‑Parameter Model Takes on Claude in Complex Tasks

GLM-5, the new 744‑billion‑parameter open‑source LLM, expands on GLM‑4.5 with GlmMoeDsa architecture, achieves higher HLE benchmark scores than Claude Opus 4.5, demonstrates strong long‑context and agent capabilities, supports vLLM/SGLang, runs on various Chinese chips, and can directly generate Office documents.

AI benchmarksChinese chipsClaude
0 likes · 5 min read
GLM-5 Unveiled: 744B‑Parameter Model Takes on Claude in Complex Tasks
HyperAI Super Neural
HyperAI Super Neural
Feb 10, 2026 · Artificial Intelligence

WeDLM Diffusion Language Model Tutorial: 3× Faster Inference Than vLLM AR Models

The Tencent WeChat AI team introduces WeDLM, a diffusion language model that, through topological reordering, surpasses autoregressive models on the industrial‑grade vLLM engine with over threefold speedup on math reasoning and up to tenfold in low‑entropy scenarios, and provides a step‑by‑step online tutorial with GPU compute credits.

Diffusion Language ModelGPU computeLarge Language Model
0 likes · 5 min read
WeDLM Diffusion Language Model Tutorial: 3× Faster Inference Than vLLM AR Models
Old Zhang's AI Learning
Old Zhang's AI Learning
Feb 3, 2026 · Artificial Intelligence

Step‑3.5‑Flash: Lightning‑Fast Inference with 196B Params, Only 11B Active (vLLM)

Step‑3.5‑Flash, a 196‑billion‑parameter open‑source LLM that activates only 11 B per token via a Mixture‑of‑Experts design, delivers 3‑plus‑times faster inference, matches top‑tier closed‑source models on SWE‑bench and other benchmarks, supports 256 K context, runs on consumer‑grade hardware, and is already integrated into vLLM, SGLang, and Claude Code, though it has known token‑efficiency and domain‑stability limitations.

LLM BenchmarkMoEStep-3.5-Flash
0 likes · 11 min read
Step‑3.5‑Flash: Lightning‑Fast Inference with 196B Params, Only 11B Active (vLLM)
Old Zhang's AI Learning
Old Zhang's AI Learning
Feb 3, 2026 · Artificial Intelligence

Why GLM-OCR Leads OCR Benchmarks: 0.9B Model Tops OmniDocBench

GLM-OCR, a 0.9B‑parameter multimodal OCR model from Zhipu, achieves the highest score (94.62) on OmniDocBench V1.5, offers lightweight deployment via vLLM, Ollama, API and SDK, and outperforms larger rivals like DeepSeek‑OCR and PaddleOCR in speed and accuracy.

DeploymentGLM-OCROCR
0 likes · 10 min read
Why GLM-OCR Leads OCR Benchmarks: 0.9B Model Tops OmniDocBench
AI Waka
AI Waka
Feb 1, 2026 · Artificial Intelligence

Boost LLM Inference Speed: Precision Tricks, Quantization, and Multi‑GPU Strategies

This article reviews practical techniques for accelerating large language model inference—including reduced‑precision formats, post‑training quantization, adapter‑based fine‑tuning, pruning, continuous batch processing, and multi‑GPU deployment—while providing concrete code examples, benchmark results, and guidance on selecting the right approach for production workloads.

GPULLMQuantization
0 likes · 20 min read
Boost LLM Inference Speed: Precision Tricks, Quantization, and Multi‑GPU Strategies
Old Zhang's AI Learning
Old Zhang's AI Learning
Feb 1, 2026 · Artificial Intelligence

Microsoft VibeVoice‑ASR Open‑Source: One‑Shot 60‑Minute Transcription with Speaker ID and Timestamps

Microsoft’s newly open‑sourced VibeVoice‑ASR model can transcribe up to 60‑minute audio in a single pass, preserving global context while providing built‑in speaker diarization and timestamps, supports 50+ languages, offers custom hot‑word injection, and can be deployed via Docker, Gradio, or vLLM for high‑throughput API services.

ASRDockerLoRA
0 likes · 9 min read
Microsoft VibeVoice‑ASR Open‑Source: One‑Shot 60‑Minute Transcription with Speaker ID and Timestamps
Old Zhang's AI Learning
Old Zhang's AI Learning
Jan 30, 2026 · Artificial Intelligence

Qwen3-ASR: Open‑Source Speech Recognition Supporting 52 Languages and Dialects, Outperforming Whisper

The Qwen3‑ASR series, now open‑sourced by Alibaba, offers three models (1.7B, 0.6B, and a 0.6B forced aligner) that cover 52 languages and 22 Chinese dialects, support streaming and offline inference, achieve an RTF of 0.064 with 2000× realtime throughput, handle singing with background music, and provide detailed deployment guides, benchmarks, and comparisons with other ASR solutions.

Qwen3-ASRReal-time inferenceforced aligner
0 likes · 15 min read
Qwen3-ASR: Open‑Source Speech Recognition Supporting 52 Languages and Dialects, Outperforming Whisper
Old Zhang's AI Learning
Old Zhang's AI Learning
Jan 28, 2026 · Artificial Intelligence

How to Deploy DeepSeek‑OCR‑2 Locally: A Hands‑On Walkthrough

The article details a step‑by‑step local deployment of DeepSeek‑OCR‑2, covering GPU memory requirements, accuracy on complex tables, long inference times, dependency hurdles like GCC, GLIBC and flash‑attn, and provides concrete solutions using conda environments and symlinks.

CondaDeepSeek-OCR 2Deployment
0 likes · 7 min read
How to Deploy DeepSeek‑OCR‑2 Locally: A Hands‑On Walkthrough
Old Zhang's AI Learning
Old Zhang's AI Learning
Jan 27, 2026 · Artificial Intelligence

Can Kimi K2.5’s Visual Agent Swarm Make It the New Open‑Source AI King?

Kimi K2.5, Moonshot’s latest open‑source multimodal model trained on 15 trillion image‑text tokens, adds native vision capabilities and a 100‑agent swarm that speeds complex tasks by 4.5×, achieves top‑tier benchmark scores, and can be deployed with vLLM, while demanding significant resources and hardware.

Agent SwarmBenchmarkKimi K2.5
0 likes · 10 min read
Can Kimi K2.5’s Visual Agent Swarm Make It the New Open‑Source AI King?
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Jan 27, 2026 · Artificial Intelligence

Deploying Qwen3 on Kunlun P800: Full‑Parameter DPO Training and Inference Guide

This guide walks through setting up a Kunlun P800 XPU host, preparing Docker containers, deploying Qwen3‑8B/‑32B/‑VL models with vLLM‑Kunlun, benchmarking performance, and running full‑parameter DPO training using LLaMA‑Factory, providing scripts, configuration files, and troubleshooting tips for AI engineers.

DPOKunlun P800LLaMA-Factory
0 likes · 32 min read
Deploying Qwen3 on Kunlun P800: Full‑Parameter DPO Training and Inference Guide
Old Zhang's AI Learning
Old Zhang's AI Learning
Jan 27, 2026 · Artificial Intelligence

DeepSeek-OCR 2 Enables AI to Read Images with Human‑Like Logical Flow

DeepSeek-OCR 2 introduces Visual Causal Flow and a LLM‑based visual encoder, achieving 91.09% accuracy on OmniDocBench v1.5, while providing detailed installation, two inference modes (vLLM and Transformers), and an analysis of its strengths and limitations for complex document processing.

DeepEncoder V2DeepSeek-OCR 2LLM
0 likes · 9 min read
DeepSeek-OCR 2 Enables AI to Read Images with Human‑Like Logical Flow
AI Cyberspace
AI Cyberspace
Jan 26, 2026 · Artificial Intelligence

How NVFP4 Quantization Supercharges LLM Inference on NVIDIA DGX

This article explains the NVFP4 4‑bit floating‑point quantization technique, shows how to deploy Qwen3‑30B‑A3B models with TensorRT‑LLM and vLLM, compares performance across NVFP4, AWQ and INT8 quantizations, and provides practical profiling commands for NVIDIA DGX systems.

LLMNVFP4NVIDIA DGX
0 likes · 23 min read
How NVFP4 Quantization Supercharges LLM Inference on NVIDIA DGX
Old Zhang's AI Learning
Old Zhang's AI Learning
Jan 24, 2026 · Artificial Intelligence

Open-Source Qwen3‑TTS: Sub‑100 ms Latency, Runs on 8 GB GPU, and ComfyUI Integration

Qwen3‑TTS, an open‑source text‑to‑speech model from Alibaba, offers sub‑100 ms first‑packet latency, supports voice cloning, natural‑language voice design, and ten languages, can be deployed locally on a GPU with as little as 8 GB VRAM, and integrates with ComfyUI for visual workflow building.

ComfyUIOpen-sourceQwen3-TTS
0 likes · 15 min read
Open-Source Qwen3‑TTS: Sub‑100 ms Latency, Runs on 8 GB GPU, and ComfyUI Integration
Old Zhang's AI Learning
Old Zhang's AI Learning
Jan 23, 2026 · Artificial Intelligence

Open‑Source GLM‑ASR‑Nano‑2512: Chinese Dialect‑Optimized Speech Recognition on Consumer‑Grade GPUs

GLM‑ASR‑Nano‑2512, a 1.5 B‑parameter open‑source speech‑recognition model released in December 2025, delivers state‑of‑the‑art accuracy on Chinese dialects and low‑volume audio, outperforms Whisper V3 on benchmark tests, runs on consumer GPUs, and provides detailed installation and deployment guides for transformers, vLLM and SGLang.

Chinese dialectsGLM-ASR-Nano-2512Open-source
0 likes · 11 min read
Open‑Source GLM‑ASR‑Nano‑2512: Chinese Dialect‑Optimized Speech Recognition on Consumer‑Grade GPUs
AI Engineering
AI Engineering
Jan 23, 2026 · Industry Insights

vLLM Core Team Launches Inferact, Secures $150M Seed Funding

The vLLM core maintainers have founded Inferact, raised a $150 million seed round led by Andreessen Horowitz and Lightspeed, and highlighted escalating inference challenges, the project's ecosystem dominance, and a continued commitment to open‑source development.

AI InfrastructureInferactLLM Inference
0 likes · 3 min read
vLLM Core Team Launches Inferact, Secures $150M Seed Funding
Fun with Large Models
Fun with Large Models
Jan 22, 2026 · Artificial Intelligence

Step‑by‑Step Guide to Calling Locally Deployed LLMs via OpenAI API Format in Python

This tutorial explains the OpenAI‑style request and response schema, demonstrates low‑level API calls with the requests library, compares them to the high‑level openai package, and walks through building a streaming multi‑turn chatbot that interacts with a locally hosted large language model.

ChatbotLarge Language ModelOpenAI API
0 likes · 17 min read
Step‑by‑Step Guide to Calling Locally Deployed LLMs via OpenAI API Format in Python
Ops Community
Ops Community
Jan 18, 2026 · Artificial Intelligence

How to Quadruple LLM Throughput with vLLM’s PagedAttention and Continuous Batching

This guide details how to replace native Transformers inference with the high‑performance vLLM engine, leveraging PagedAttention, continuous batching, tensor parallelism, and OpenAI‑compatible APIs to achieve 3‑4× higher throughput, lower latency, and scalable multi‑GPU deployments for production‑grade large language models.

Continuous BatchingGPU OptimizationOpenAI API Compatibility
0 likes · 61 min read
How to Quadruple LLM Throughput with vLLM’s PagedAttention and Continuous Batching
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Jan 12, 2026 · Artificial Intelligence

How to Reduce Large‑Model Inference Cold‑Start to Seconds with vLLM Optimizations

This article details how Baidu Cloud's hybrid‑cloud team leveraged the vLLM framework to cut the cold‑start time of massive models like Qwen3‑235B‑A22B from minutes to a few seconds through accelerated weight loading, CUDA‑graph capture postponement, cross‑instance state reuse, fork‑based process startup, and guard‑instance pre‑warming techniques.

CUDA Graphcold-start optimizationlarge model inference
0 likes · 16 min read
How to Reduce Large‑Model Inference Cold‑Start to Seconds with vLLM Optimizations
Baidu Geek Talk
Baidu Geek Talk
Jan 7, 2026 · Artificial Intelligence

How Baidu’s vLLM‑Kunlun Plugin Powered MiMo Flash V2 on Kunlun XPU in 2 Days

Within two days, Baidu’s Baige and Kunlun Chip teams adapted the 309‑billion‑parameter MiMo Flash V2 model—featuring a hybrid SWA+Sink and Full Attention mechanism—to run efficiently on the Kunlun P800 XPU using the vLLM‑Kunlun Plugin, achieving lossless performance comparable to GPU inference.

AI inferenceKunlun XPUMiMo Flash V2
0 likes · 7 min read
How Baidu’s vLLM‑Kunlun Plugin Powered MiMo Flash V2 on Kunlun XPU in 2 Days
58 Tech
58 Tech
Jan 6, 2026 · Artificial Intelligence

How vLLM 0.8.4 Implements Multi‑LoRA for Efficient Large‑Model Inference

This article provides a step‑by‑step technical walkthrough of vLLM 0.8.4 on a single GPU, detailing the platform’s startup, model loading, Multi‑LoRA deployment, internal ZMQ communication, request scheduling, and inference execution, while exposing key source‑code snippets and architectural diagrams.

GPU inferenceLoRA adaptersMulti-LoRA
0 likes · 35 min read
How vLLM 0.8.4 Implements Multi‑LoRA for Efficient Large‑Model Inference
Ops Community
Ops Community
Dec 28, 2025 · Artificial Intelligence

Boost LLM Inference Speed: Build a High‑Concurrency vLLM Service with Best‑Practice Ops

This guide walks through the complete process of deploying a high‑throughput large language model inference service using vLLM, covering environment preparation, installation, configuration tuning, performance testing, real‑world case studies, monitoring, troubleshooting, and backup strategies for production‑grade deployments.

DeploymentGPU OptimizationHigh concurrency
0 likes · 44 min read
Boost LLM Inference Speed: Build a High‑Concurrency vLLM Service with Best‑Practice Ops
MaGe Linux Operations
MaGe Linux Operations
Dec 26, 2025 · Operations

Taming vLLM OOM: Real‑World Causes and Proven Fixes for Production

This article examines why vLLM experiences out‑of‑memory errors in production, explains memory fragmentation caused by PagedAttention, outlines four typical OOM scenarios with concrete command‑line solutions, and provides deep analysis, configuration scripts, dynamic tuning, troubleshooting flowcharts, monitoring alerts, and best‑practice recommendations.

DeploymentGPUMemory Fragmentation
0 likes · 24 min read
Taming vLLM OOM: Real‑World Causes and Proven Fixes for Production
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Dec 22, 2025 · Artificial Intelligence

The Core Ideas Behind Paged Attention for KV‑Caching

This article explains how Paged Attention, introduced by the vLLM team, applies virtual‑memory techniques, non‑contiguous block mapping, copy‑on‑write reuse, distributed scheduling, and hardware‑level optimizations to improve KV‑cache efficiency and reduce memory fragmentation in large language model serving.

Copy-on-WriteDistributed SchedulingGPU Memory Management
0 likes · 6 min read
The Core Ideas Behind Paged Attention for KV‑Caching
Alibaba Cloud Infrastructure
Alibaba Cloud Infrastructure
Dec 22, 2025 · Artificial Intelligence

Boost LLM Inference with KV‑Cache‑Aware Routing on Alibaba Cloud ACK GIE

This article explains why KV‑Cache hit rate is critical for large‑model inference, describes vLLM's automatic prefix caching, outlines the distributed cache challenges, and provides a step‑by‑step guide to deploying Alibaba Cloud ACK Gateway with Inference Extension's precise‑mode prefix‑cache‑aware routing, backed by benchmark results.

Alibaba CloudKV cacheKubernetes
0 likes · 18 min read
Boost LLM Inference with KV‑Cache‑Aware Routing on Alibaba Cloud ACK GIE