BladeLLM: Ultra‑Long Context LLM Inference via RaggedAttention & AutoTuner

Background Long‑context capability is becoming essential for large language models (LLMs) as it enables applications such as personalized chatbots, literary generation, and document summarization. Existing inference engines struggle with the memory and compute demands of very long sequences.

BladeLLM Overview BladeLLM is Alibaba Cloud’s LLM inference platform that aims to provide high‑performance, low‑cost LLM services. It optimizes the full inference stack and, notably, extends the maximum supported context length far beyond typical limits.

Technical Solutions

RaggedAttention Inspired by TensorFlow’s RaggedTensor, RaggedAttention stores each sequence’s key/value cache with variable length while keeping each sequence’s cache contiguous in memory. This design improves memory‑access efficiency compared with PagedAttention, at the cost of slightly higher fragmentation, offering a practical trade‑off for diverse workloads.

DNN‑based AutoTuner In dynamic‑shape inference scenarios, BladeLLM replaces costly runtime tuning with a deep‑neural‑network predictor that selects the optimal kernel schedule without actual measurement. The predictor achieves 99.39% of the performance of exhaustive tuning while reducing prediction latency to ~2 µs and using only a single CPU core, avoiding interference with GPU inference.

Performance Results

Benchmarks show BladeLLM supporting up to 70 K tokens (potentially 280 K with KV‑cache quantization) while maintaining low latency. Compared with other systems (lmDeploy, vLLM, HuggingFace Llama, LightLLM), BladeLLM delivers higher throughput and avoids hangs or OOM failures at long lengths.

Figures illustrate the token generation time growth for Llama‑2‑13B and the performance advantage of the AutoTuner over PyTorch, TorchScript, and ONNX Runtime.

Conclusion

Ultra‑long context is a critical trend for LLMs, and BladeLLM’s RaggedAttention and DNN‑based AutoTuner provide a scalable solution that balances memory efficiency and compute speed. Future work will continue to explore quantization, multi‑turn dialogue, and kernel optimizations.