Tagged articles
SGLang
48 articles · Page 1 of 1
Researchers from Tsinghua and CUHK propose Cache Engineering, a paradigm where agents directly manipulate KV cache via Append, Drop, and Repos operations, avoiding re-prefill overhead; their open-source AKV system reduces average inference cost by 27.7% and improves task success rate by 1.7% across four agent benchmarks.
AKVAgentCache Engineering
0 likes · 13 min read

FlashStates introduces arbitrary state checkpoints and cost-aware eviction for hybrid LLM architectures, achieving up to 3.14x prefill throughput and 88% TTFT reduction on real agent workloads, with CPU cache integration and cross-engine reuse.
FlashStatesGreedyDual-SizeKV Cache
0 likes · 13 min read

Researchers from Tencent Youtu Lab, Shanghai Jiao Tong University, and Xiamen University introduce DSpine, a parallel speculative decoding method that embeds causal dependencies layer-by-layer in the draft network, achieving state-of-the-art acceptance length and up to 4x throughput on SGLang, with open-source code and models.
DSpineLLM inference accelerationSGLang
0 likes · 17 min read

A comprehensive comparison of 15 local LLM inference frameworks on Apple Silicon, organized into kernel engines (llama.cpp, MLX), desktop wrappers (Ollama, LM Studio, Jan), and service/cluster tools (SGLang, vllm-mlx, exo), with real benchmarks and a workflow-based decision flowchart.
Apple SiliconExoLLM inference
0 likes · 12 min read

vLLM and SGLang matched or exceeded DeepSeek's custom DeepInfer engine on DeepSeek-V4.1-Flash within days by adopting its open-sourced operators and optimizing for model-specific traits, proving general-purpose inference engines remain dominant due to full production-grade systems.
DeepInferDeepSeekKV Cache
0 likes · 15 min read

This comprehensive guide explains inference frameworks for large language models, covering core concepts like Prefill/Decode and KV Cache, comparing major open-source frameworks (vLLM, SGLang, TensorRT-LLM), detailing request processing flow from scheduling to sampling, and identifying who benefits from understanding inference infrastructure.
KV CacheLLM servingMoE
0 likes · 46 min read

Baidu Baige presents a three-layer stability framework for LLM inference services: tiered traffic control with minute-level autoscaling, dynamic request bucketing with cross-bucket KVCache migration boosting decode throughput 40-50%, and multi-level KV storage with global cache awareness keeping hit rates above 90% of theoretical maximum.
KVCacheLLM inferenceMulti-level Cache
0 likes · 21 min read

Baidu Baige presents extreme VRAM optimization for TB-scale model inference: zero-redundancy weight deployment via DP Attn-sharding (releasing 16–30 GiB/card) and full-route CPP (50% throughput gain), plus KV Cache compression via LU-KV (80% reduction, 0.52% accuracy loss), Draft SWA, and PD handshake optimization (20% VRAM release).
KV Cache compressionLLM inferenceLU-KV
0 likes · 27 min read

Xiaohongshu built a custom inference engine GR-Inference for generative search retrieval, addressing unique load characteristics — long context, short decode, large dynamic beam, and constrained generation — that break general frameworks, achieving 1.5–3.6x throughput over SGLang and improving recall and click-through rates.
GR-InferenceInference OptimizationLLM serving
0 likes · 7 min read

This article details SGLang's device plugin mechanism (Platform and Hook) enabling hardware-agnostic inference, the SGLang-Kunlun plugin's cuda-like compatibility layer, a five-step standardized adaptation process with measurable acceptance criteria, and AI-driven Harness automation that reduced DeepSeek V4 Flash version upgrades to two hours while achieving 1.55x operator speedups via dual-stream vector/matrix compute overlap.
AI-assisted developmentCUDA GraphDevice Plugin
0 likes · 22 min read

This article traces OpenAI's API evolution across six generations from Completions to Responses API, explains why the open-source ecosystem remains anchored to Chat Completions despite official advances, compares architectural trade-offs between stateless and stateful paradigms, and provides a practical selection guide for engineers building heterogeneous model gateways or agent systems.
API EvolutionAgent ArchitectureChat Completions
0 likes · 28 min read


AntTechSep 15, 2026 · Artificial Intelligence
SingProbe introduces endogenous runtime guardrails that integrate risk detection directly into LLM inference by reusing hidden states, achieving comparable safety and hallucination detection to external guardrails while adding less than 0.5% decode overhead across 29 open-source models.
LLM guardrailsSGLangSingProbe
0 likes · 25 min read

This article explains why GPUs underutilize during LLM inference and compares static, dynamic, and continuous batching strategies, detailing their trade-offs, the role of KV cache, iteration-level scheduling, and chunked prefill to optimize throughput for variable-length outputs.
GPU utilizationKV CacheLLM inference
0 likes · 13 min read

This article distinguishes between inference frameworks (llama.cpp, vLLM, SGLang) that execute model computations and inference platforms (Ollama, LM Studio, Xinference) that manage deployment, explaining their roles, interactions, and how to choose tools for local or server-side LLM inference.
AI inferenceLLM servingLM Studio
0 likes · 16 min read

Low GPU utilization and mysterious memory growth often stall AI workloads, but existing tools are intrusive and limited; SysOM AI Profiling offers zero‑intrusion, one‑click, multi‑dimensional analysis across training and inference, with timeline visualisation, memory‑snapshot, and iteration‑anchored diff, demonstrated by vLLM memory‑leak detection and SGLang vs vLLM performance comparison.
AI profilingGPU utilizationMemory Leak
0 likes · 11 min read

The article shows that in AI‑assisted coding the real cost driver is cache hit rate, not model intelligence, using a lawyer‑memory analogy, hierarchical caching details, three SGLang PR fixes, token‑usage statistics, benchmark scores, and four concrete task‑level observations that together explain how Agnes 2.5 Pro Alpha reduces latency and expense.
AI programmingAgnesAISGLang
0 likes · 22 min read

This guide walks through deploying SGLang on Linux with NVIDIA GPUs and Docker Compose, covering environment checks, image versioning, minimal foreground launch, Docker Compose configuration, health checks, troubleshooting, performance testing, security hardening, and upgrade/rollback procedures to reliably expose an OpenAI‑compatible large model API in production.
APIDocker ComposeGPU
0 likes · 24 min read

Alibaba Mama AI Infra and AMD collaborated to integrate SGLang‑ATOM plugin mode for Qwen 3.5/3.6 models on MI308X GPUs, detailing multimodal, GDN, KV‑cache, and parallel‑strategy optimizations that achieve low‑latency, high‑throughput inference suitable for complex advertising workloads.
AMD ATOMGPU optimizationLLM inference
0 likes · 18 min read

The SGLang team turned their benchmarking, profiling, CUDA kernel tuning, and production‑issue triage know‑how into reusable agent skills, merging three KDA‑Pilot PRs that delivered up to 2.75× kernel acceleration, a 71.4% throughput boost for Qwen3‑Next and a TTFT reduction from 456 ms to 168 ms, while outlining a repeatable workflow and practical rules for large‑scale performance engineering.
Agent AutomationCUDA optimizationLLM serving
0 likes · 16 min read

Semgrep’s benchmark shows open‑source GLM‑5.2 matching Claude’s performance while costing only $0.17 per vulnerability, and Nvidia’s NVFP4 quantization halves the model’s memory footprint with virtually unchanged accuracy, making local deployment on 8‑GPU systems far more affordable.
AI DeploymentGLM-5.2NVFP4
0 likes · 11 min read

OSCAR presents an attention‑aware rotation scheme that compresses KV caches to true 2‑bit, cutting memory usage by up to 8× and boosting decode throughput by up to 7×, while preserving inference quality within a few points of BF16 across multiple models and long‑context benchmarks, outperforming TurboQuant.
2-bit QuantizationKV CacheLong-context inference
0 likes · 13 min read

OSCAR, a new attention‑aware 2‑bit KV cache quantization method, cuts KV memory by up to 8×, delivers up to 3× decode speedup and 7× throughput gain, and matches BF16 accuracy across 4B‑32B models on diverse long‑context reasoning tasks, surpassing TurboQuant.
2-bit compressionKV CacheLLM quantization
0 likes · 12 min read

MiMo announces a permanent up to 99% price reduction for its V2.5 API, a 5‑8× usage boost in its Token Plan billing, a full reset of all Token Plan quotas, and the conclusion of its Hundred‑Trillion Token Creator Incentive Program, effective May 27, 2026.
AI infrastructureAPI pricingInference Optimization
0 likes · 5 min read

The recent SGLang × MUSA meetup revealed that MUSA’s GPU backend has been merged into SGLang’s official codebase, delivering zero‑learning‑cost integration, performance gains of up to 66 % on DeepSeek‑V4, and a growing ecosystem of adapters, high‑performance kernels, and distributed inference support.
AI inferenceDeepSeekGPU
0 likes · 12 min read

Ant Group's Ling‑2.6‑1T, a 1‑trillion‑parameter LLM built for token efficiency and fast‑thinking, outperforms on elite reasoning and agentic benchmarks, offers easy local deployment via vLLM or SGLang, provides a quantized 3.6‑bit version, and includes practical usage tips for developers and knowledge workers.
Agentic ModelClaude Code IntegrationLing-2.6-1T
0 likes · 12 min read

RadixArk raised a $100 million seed round backed by top hardware and AI investors to turn the open‑source SGLang inference engine and the Miles RL framework into day‑0 standards, aiming to democratize AI infrastructure and eliminate bottlenecks from training to inference.
AI infrastructureHardware‑agnostic AIOpen-source inference
0 likes · 10 min read

The article analyzes DeepSeek‑V4's architectural innovations—including mixed sparse attention, mHC, and native FP4 weights—explains SGLang's ShadowRadix, HiSparse, and in‑graph speculative decoding solutions, presents benchmark gains, provides Docker deployment steps, and warns of key pitfalls for long‑context inference.
HiSparseLong-context inferenceSGLang
0 likes · 15 min read

Kimi K2.6, an open-source 1-trillion-parameter MoE model, expands Agent capabilities with 256K context, multimodal inputs, and the ability to coordinate 300 sub-Agents over 4,000 steps, achieving top scores on benchmarks like Terminal-Bench 2.0, SWE-Bench Pro, and BrowseComp, while offering flexible deployment via vLLM, SGLang, and KTransformers.
KTransformersKimi K2.6SGLang
0 likes · 11 min read

The DFlash approach replaces speculative decoding’s autoregressive drafter with a block diffusion model and injects target‑model hidden features into every KV‑cache layer, achieving up to 5× speed‑up for Qwen3.5‑27B on single‑GPU and 1.5–1.9× on high‑concurrency workloads while preserving output quality.
DFlashQwen3.5SGLang
0 likes · 12 min read

MiniMax‑M2.7, the newly open‑sourced 230‑billion‑parameter MoE model, offers self‑evolution, professional software engineering and agent capabilities, and can be deployed locally using Ollama, vLLM, SGLang or Docker with 4‑8 H200 GPUs, while the article details hardware needs, performance gains and tool‑calling/Thinking features.
GPULLMMiniMax-M2.7
0 likes · 11 min read

This article reviews the DeepLearning.ai short course on SGLang, explains why large‑language‑model inference is slow, details how KV Cache reduces the computation from O(n²) to O(n), introduces RadixAttention for cross‑request caching, and presents code examples and benchmark results showing up to 10× speedup in real‑world RAG scenarios.
KV CacheLLM inferencePython
0 likes · 13 min read

The article examines the hype around “harness engineering” in LLM workflows, showing through SGLang’s multi‑agent experience that the approach merely repackages established software‑engineering principles such as separation of concerns, docs‑as‑code, and structured routing, and discusses its limits and future implications.
Harness EngineeringLLMSGLang
0 likes · 8 min read

The article traces five eras of KV cache management for LLM inference—from its absence before Transformers to the emerging unified hybrid memory architecture—comparing vLLM, SGLang, and TensorRT‑LLM and offering a decision framework for selecting the right solution in various deployment scenarios.
KV CacheLLM inferencePagedAttention
0 likes · 16 min read

This guide walks through preparing the Kimi 2.5 model, uploading it to OSS, configuring persistent storage, and using SGLang, RoleBasedGroup, and Openclaw to deploy a production‑grade inference service on Alibaba Cloud Kubernetes with step‑by‑step commands and YAML examples.
AIKimiKubernetes
0 likes · 14 min read

GLM‑ASR‑Nano‑2512, a 1.5 B‑parameter open‑source speech‑recognition model released in December 2025, delivers state‑of‑the‑art accuracy on Chinese dialects and low‑volume audio, outperforms Whisper V3 on benchmark tests, runs on consumer GPUs, and provides detailed installation and deployment guides for transformers, vLLM and SGLang.
Chinese dialectsGLM-ASR-Nano-2512SGLang
0 likes · 11 min read

SGLang, the open‑source inference accelerator, has been spun out into RadixArk—a $400 million‑valued startup aiming to democratize AI infrastructure, while the broader market sees a surge of funding for inference‑focused companies.
AI inferenceAI infrastructureRadixArk
0 likes · 5 min read

This guide details how switching from vLLM to SGLang on eight NVIDIA H800 GPUs increased Llama‑3‑70B‑Instruct throughput from 180 to 420 tokens per second, covering SGLang’s core innovations, environment setup, configuration tweaks, performance benchmarks, troubleshooting tips, and production‑grade deployment scripts.
FlashInferGPU optimizationH800
0 likes · 19 min read

This article explains how SGLang’s hybrid model design combines Transformer attention with Mamba state‑space layers, introduces a dual‑pool memory architecture and elastic allocation, and presents specialized prefix‑cache and speculative‑decoding techniques that together enable efficient, scalable inference for long‑context large language models.
Hybrid ModelsInference OptimizationKVCache
0 likes · 22 min read

The article details the Attention‑FFN Disaggregation (AFD) technique used by Baidu Baige to separate self‑attention and feed‑forward network stages in DeepSeek‑V3 models, describing multi‑stage scheduling, three‑batch overlap, communication optimizations, and performance results that achieve up to 19% throughput improvement under a 100 ms SLO.
3BOAFDAttention-FFN Disaggregation
0 likes · 17 min read

Token‑level Two‑Chunk Overlap replaces traditional batch‑level Two‑Batch Overlap, dynamically splitting sequences into balanced token chunks, enabling near‑equal compute and communication times, improving GPU utilization and achieving up to 30% throughput gains in heterogeneous request workloads, with zero accuracy loss.
GPU utilizationLLM inferenceSGLang
0 likes · 9 min read

LongCat-Flash, an open‑source Mixture‑of‑Experts model released by Meituan, leverages model‑system co‑design, PD‑disaggregation, SBO scheduling and large‑scale expert parallelism within the SGLang framework to deliver dramatically lower latency, higher throughput and cost‑effective inference for AI agents, with detailed deployment instructions provided.
LongCat-FlashMixture of ExpertsSGLang
0 likes · 15 min read

This article examines how Volcano Engine's Elastic Instant Cache (EIC) tackles the memory bottleneck, high‑concurrency latency, and cross‑node coordination challenges of large language model inference by decoupling storage and computation, pooling resources, and applying layered optimizations, ultimately boosting AI inference efficiency, scalability, and cost‑effectiveness across various deployment scenarios.
AI infrastructureDistributed CachingKVCache
0 likes · 30 min read

This article shares a detailed, first‑hand analysis of the technical challenges, framework choices, memory management, weight conversion, precision alignment, and efficiency optimizations encountered while building reinforcement‑learning pipelines for a 671‑billion‑parameter model using the verl ecosystem.
GPU memory managementMegatronRL training
0 likes · 16 min read

This article provides a comprehensive comparison of seven popular large‑language‑model inference engines—Transformers, vLLM, Llama.cpp, SGLang, MLX, Ollama and others—detailing their core features, performance characteristics, hardware compatibility, concurrency support, and ideal use‑cases, plus practical installation guidance for Xinference.
LLMMLXSGLang
0 likes · 17 min read

This guide shows how to use the lightweight OpenStation platform to install, configure, and launch the DeepSeek‑R1 large‑model on a personal server in under 15 minutes, covering zero‑code deployment, resource management, inference engine selection, and integration with CherryStudio.
AI model deploymentCherryStudioDeepSeek-R1
0 likes · 7 min read

The article summarizes Zhihu’s technical talk on the ZhiLight large‑model inference framework, detailing model execution mechanisms, GPU load analysis, multi‑GPU parallel strategies, open‑source engine comparisons, compute‑communication overlap, quantization techniques, benchmark results, and future directions for scalable LLM deployment.
GPU ParallelismSGLangZhiLight
0 likes · 11 min read

Meituan’s search and recommendation team converted the FP8‑only DeepSeek‑R1 model to INT8 by first casting weights to BF16 and then applying block‑wise or channel‑wise quantization, which preserves GSM8K and MMLU accuracy while delivering 33% to 50% higher throughput on A100‑80G GPUs, and they released the SGLang‑based inference scripts and quantized weights publicly, enabling deployment on older NVIDIA hardware without accuracy loss.
DeepSeek-R1GPU deploymentINT8 quantization
0 likes · 11 min read

The Inspur Metabrain R1 inference server, equipped with FP8 acceleration and a 1128 GB HBM3e memory pool, has been tightly integrated with SGLang 0.4.3 to run the 671‑billion‑parameter DeepSeek R1 model, delivering over 1,000 concurrent user sessions and up to 3,976 tokens/s throughput.
AI serverDeepSeekInference Optimization
0 likes · 5 min read
