How Llumnix Cuts LLM Serving Latency by 10× with Dynamic Scheduling

Llumnix Accepted at OSDI ’24

Alibaba Cloud’s AI platform PAI has had its paper “Llumnix: Dynamic Scheduling for Large Language Model Serving” accepted at the prestigious OSDI 2024 conference.

What is Llumnix?

Llumnix is the first industry framework that can dynamically re‑allocate inference requests among multiple LLM instances at runtime. By exploiting request‑level dynamism, it provides load balancing, fragmentation mitigation, priority handling and other scheduling optimizations.

Performance Gains

Experiments on LLaMA‑family models show that Llumnix reduces tail latency by more than 10×, speeds up high‑priority requests by 1.5×, and cuts service cost to 64 % of the baseline.

Open‑Source Release

The system is open‑sourced on GitHub ( https://github.com/AlibabaPAI/llumnix) and currently supports vLLM as the backend inference engine, automatically launching multiple vLLM instances and performing runtime request rescheduling.

Future Integration

Future versions will tightly integrate with Alibaba Cloud’s BladeLLM inference engine, PAI‑EAS model‑as‑a‑service, and the PAI Lingjun intelligent computing service to form a unified high‑performance LLM serving suite.

Paper Details

Title: Llumnix: Dynamic Scheduling for Large Language Model Serving Authors: Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, Wei Lin Link: https://www.usenix.org/conference/osdi24/presentation/sun-biao