DeepSeek Open-Sources DeepInfer: Inference Framework Reveals 82% Gross Margin on AI Serving

DeepSeek Open-Sources DeepInfer Inference Framework

On October 20, DeepSeek released its internal inference framework DeepInfer and the companion token encoding/decoding component deepseek-recipe under the Apache 2.0 license. DeepInfer currently supports the DeepSeek-V4.1-Flash model and is the foundation of DeepSeek's large-scale inference service.

Economics: 155 Trillion Tokens/Day, $38/GPU/Hour Revenue, 82% Gross Margin

DeepSeek disclosed that a single cluster serves 155 trillion tokens per day . At official API list prices, this translates to $38 per GPU per hour in revenue. Assuming a GPU rental cost of $7 per hour for a B300 (the prevailing cloud rate), the gross margin works out to approximately 82% . This figure sparked widespread discussion on Hacker News, where one commenter described it as "a nuclear bomb" — cold, hard numbers suggesting AI inference is not a bubble.

Technical Architecture of DeepInfer

Hardware support: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

Core optimizations: Fused operators, distributed parallelism, speculative decoding, asynchronous scheduling, and JSON Schema constrained output.

Lineage: Initially built on vLLM, with references to SGLang and TensorRT-LLM.

The companion deepseek-recipe handles conversion between various API request formats (Chat Completions, Responses, Messages) and the model's prompt/token sequences, including reasoning traces, tool calls, and streaming output. This reduces behavioral drift when the same model is deployed on third-party infrastructure.

Deployment Gaps: What's Still Needed for Production

The open-source repositories provide offline inference and core large-scale serving capabilities. A complete online service still requires integration of:

KV Cache storage systems

Vision encoders

Prefill/Decode disaggregated deployment

Cross-replica load balancing

The overarching goal is to minimize redundant computation per GPU and maximize request throughput.

Key Optimization Strategies from DeepSeek's Production Deployment

Prefill/Decode Disaggregation

Requests are split into the Prefill phase (processing input context) and the Decode phase (generating tokens). Separating these stages allows independent resource allocation, avoiding contention between different workload profiles and simultaneously optimizing time-to-first-token and per-token latency.

Long-Term KV Cache Retention

DeepSeek retains KV caches for 3 days for agent traffic and 7 days for chat traffic , stored in a dedicated 3FS storage cluster accessible by all Prefill nodes. By combining cross-day retention, shared storage, and automatic cache matching, previously computed long contexts are reused across many subsequent requests. For comparison, Anthropic, OpenAI, and Google typically retain caches for only 5 minutes to 24 hours .

DSpark Speculative Decoding with Dynamic Verification

DeepSeek's DSpark speculative decoding generates candidate tokens and verifies them with the main model. Crucially, it schedules verification based on the predicted acceptance rate of candidates, reducing compute spent on low-yield candidates. Official deployment Q&A estimates that near the listed production operating point, dynamic-length verification improves throughput by 4–7% over fixed 5-token verification.

Revenue Validation and Industry Implications

The Information reported DeepSeek's annualized revenue at ~$1 billion (~$2.74 million/day). At $38/GPU/hour, this implies roughly 3,000 GPUs in operation. The 82% margin figure aligns with earlier claims of recovering full hardware costs within 10 months. Hacker News debate continues: some argue for even higher margins using lower GPU costs, while others stress the need to amortize the model's entire lifecycle investment.

Outlook

DeepInfer provides a code starting point for optimized deployment. For most users, intensified inference competition promises lower API costs and better response latency. Whether the community adopts this framework widely or waits for the upcoming V4.1 Pro release remains to be seen.

References: Zhihu analysis: https://zhuanlan.zhihu.com/p/2095560295930508953 Hacker News discussion: https://news.ycombinator.com/item?id=50163277 The Information: DeepSeek annualized revenue hits $1B: https://www.theinformation.com/articles/deepseeks-annualized-revenue-hits-1-billion-startup-finalizes-7-5-billion-fundraising X post by Yuchenj_UW: https://x.com/Yuchenj_UW/status/2112591074409742733