Tagged articles
LLM serving
12 articles · Page 1 of 1
This comprehensive guide explains inference frameworks for large language models, covering core concepts like Prefill/Decode and KV Cache, comparing major open-source frameworks (vLLM, SGLang, TensorRT-LLM), detailing request processing flow from scheduling to sampling, and identifying who benefits from understanding inference infrastructure.
Cluster OrchestrationKV cacheLLM serving
0 likes · 46 min read

This article analyzes TokenRouter, a NeurIPS 2026 system that enables efficient token-level LLM routing through a request-centric programming interface, asynchronous model handoffs with state preservation, and throughput-optimized delayed batching, demonstrating 2-64x throughput gains over baselines across multiple routing algorithms and workloads.
LLM servingMarkov chain modelNeurIPS 2026
0 likes · 30 min read

Xiaohongshu built a custom inference engine GR-Inference for generative search retrieval, addressing unique load characteristics — long context, short decode, large dynamic beam, and constrained generation — that break general frameworks, achieving 1.5–3.6x throughput over SGLang and improving recall and click-through rates.
GR-InferenceLLM servingSGLang
0 likes · 7 min read

This article distinguishes between inference frameworks (llama.cpp, vLLM, SGLang) that execute model computations and inference platforms (Ollama, LM Studio, Xinference) that manage deployment, explaining their roles, interactions, and how to choose tools for local or server-side LLM inference.
AI inferenceLLM servingLM Studio
0 likes · 16 min read

Janus, a Service‑Engine co‑design system built on Oxygen xLLM, uses dual‑timescale scheduling, performance‑oracle‑driven placement, and a three‑state model lifecycle to handle bursty traffic, power‑law application hotness, and heterogeneous resource demands, achieving 0.97–1.0 SLO rates while cutting device usage by 27%.
AI infrastructureLLM servingSOSP 2026
0 likes · 19 min read

The SGLang team turned their benchmarking, profiling, CUDA kernel tuning, and production‑issue triage know‑how into reusable agent skills, merging three KDA‑Pilot PRs that delivered up to 2.75× kernel acceleration, a 71.4% throughput boost for Qwen3‑Next and a TTFT reduction from 456 ms to 168 ms, while outlining a repeatable workflow and practical rules for large‑scale performance engineering.
Agent AutomationCUDA optimizationLLM serving
0 likes · 16 min read

This article analyzes why vLLM's PagedAttention can cause GPU memory fragmentation and out‑of‑memory errors in production, presents four typical OOM scenarios, and provides concrete diagnostics, configuration tweaks, code examples, and monitoring strategies to eliminate the problem.
CUDAGPU memoryLLM serving
0 likes · 22 min read

This guide explains how to unleash vLLM’s full potential by optimizing batch size, leveraging 4‑bit quantization, tuning concurrency parameters, planning capacity with token‑per‑second metrics, and implementing robust monitoring to balance latency, cost, and scalability in production deployments.
LLM servingQuantizationbatching
0 likes · 10 min read

This guide walks you through deploying the DeepSeek‑R1‑Distill model on Volcengine CPU ECS instances, covering use‑case scenarios, recommended server types, Docker setup, environment configuration, and verification steps to achieve cost‑effective, high‑compatibility AI inference.
AI model deploymentCPU inferenceDeepSeek
0 likes · 6 min read

The author shares a candid half‑year reflection on entering computer‑science research, outlining practical steps for discovering research ideas, navigating papers, focusing on LLM serving systems, and emphasizing collaboration to help newcomers succeed in academia.
LLM servingSystem Designacademic journey
0 likes · 9 min read

The article introduces NanoFlow, a novel service framework that leverages intra‑device parallelism, operation‑based pipelining, and async scheduling to significantly improve large language model serving throughput, achieving up to 1.91× higher performance while integrating with Alibaba Cloud PAI.
Alibaba Cloud PAIGPU schedulingLLM serving
0 likes · 7 min read

Alibaba Cloud's PAI team unveiled Llumnix, a dynamic scheduling framework for large language model serving that dramatically reduces tail latency, speeds high‑priority requests, and cuts costs, earning acceptance at OSDI 2024 and now open‑sourced on GitHub.
AI systemsCost ReductionLLM serving
0 likes · 5 min read
