DeepSeek Open-Sources DeepInfer Inference Engine & deepseek-recipe API Adapter

DeepSeek Open-Sources DeepInfer Inference Engine

DeepSeek has released DeepInfer , a high-performance inference framework designed for large-scale production environments. The framework currently supports DeepSeek V4.1 Flash and is already serving 155 trillion tokens per day in a single cluster.

Core Challenges in LLM Inference

LLM inference consists of two stages: Prefill (processing user context) and Decode (generating tokens sequentially). Under heavy load with long contexts (hundreds of thousands of tokens) and multi-turn tool calls, the system faces queueing, device allocation, and KV Cache storage pressure.

Key Optimizations in DeepInfer

Distributed Inference & Load Balancing : Supports attention data parallelism, context parallelism, and MoE expert parallelism with expert load balancing. For V4.1 Flash, Prefill uses 2–8-way expert parallelism while Decode scales to 64–128-way expert parallelism to maximize GPU utilization.

Speculative Decoding via DSpark : Integrates DSpark with a confidence-based verification scheduler. The drafter predicts 5 tokens; a Confidence Head estimates acceptance probabilities, allocating verification compute to high-confidence candidates. Dynamic verification length improves throughput by 6% over fixed 5-token verification.

SWA Bounded Replay for Long Contexts : Uses Sliding Window Attention (SWA) to avoid persisting short-lived KV Cache. On cache miss, only the most recent 128 tokens are recomputed to restore approximate cache state, eliminating repeated full-context computation.

Performance Metrics (V4.1 Flash)

Online: Average first-token latency 400 ms, generation speed 230 tokens/s (300 tokens/s for code). Input cache hit rate 96%, per-GPU throughput 1.5M tokens/s (including cache hits).

Offline: Prefill peak ~86K tokens/s per GPU. Decode at ~100 tokens/s per user yields ~26K tokens/s per GPU aggregate.

deepseek-recipe: Production-Grade Token Encoding/Decoding

Complementing DeepInfer, deepseek-recipe handles API protocol adaptation. It performs three steps:

Unify input formats : Converts Chat Completions, Responses, and Messages APIs into a single internal conversation representation.

Encode to model tokens : Organizes text, roles, thinking mode, tool calls, and multimodal content into token sequences the model expects.

Parse and format outputs : Transforms model outputs (tool calls, thinking traces) into API-compliant response structures.

Together, DeepInfer optimizes compute efficiency while deepseek-recipe ensures API compatibility, enabling deployments that match official API behavior.

References

DeepInfer GitHub: https://github.com/deepseek-ai/DeepInfer

deepseek-recipe GitHub: https://github.com/deepseek-ai/deepseek-recipe

Technical report (Zhihu): https://zhuanlan.zhihu.com/p/2095560295930508953

Code example

[1]https://zhuanlan.zhihu.com/p/2095560295930508953
[2]https://github.com/deepseek-ai/DeepInfer
[3]https://github.com/deepseek-ai/deepseek-recipe