DeepSeek DeepInfer: How 2,400 GPUs Earn $38/Hour via SSD Cache Offload

DeepSeek recently open-sourced its large-scale inference framework DeepInfer alongside a detailed technical report titled "DeepSeek V4.1 Flash Large-Scale Inference Deployment Practice." The report discloses production metrics: a single cluster handles 155 trillion tokens per day (≈1.8 billion tokens/second) while generating an average revenue of $38 per GPU per hour at off-peak pricing.

Cluster Size Estimation from Throughput Data

The report does not state the total GPU count directly but provides a table of per-GPU throughput for Prefill and Decode stages. DeepSeek separates Prefill (prompt processing) and Decode (token generation) onto different machines: Prefill runs on common 8-GPU servers, Decode on 64–128 GPU super-nodes. A footnote notes the deployed Prefill-to-Decode GPU ratio is approximately 1:1.

Using the table values:

One Prefill GPU processes 1.5M tokens/second (with cache hits) → 129.6 billion input tokens/day.

One Decode GPU generates 14K tokens/second → 1.21 billion output tokens/day.

One Prefill+Decode pair (2 GPUs) yields ~130.8 billion tokens/day (0.1308 trillion).

Dividing the cluster total of 155 trillion tokens/day by 0.1308 trillion gives ~1,185 pairs, i.e., ~2,370 core compute GPUs. Adding ~20% headroom for traffic spikes brings the cluster to roughly 2,400–3,000 GPUs. The DeepInfer repository confirms the hardware: NVIDIA SM100 (Blackwell) GPUs, ~1,200 Prefill GPUs in 8-GPU servers (~150 nodes) and ~1,200 Decode GPUs in 64/128-GPU super-nodes (~18–19 super-nodes). Decode throughput reaches 230 tokens/second per user, 300 tokens/second for coding workloads.

Revenue Breakdown at Off-Peak Pricing

DeepSeek V4.1 Flash API pricing (off-peak): cache-hit input ¥0.02/M tokens, cache-miss input ¥1/M tokens, output ¥4/M tokens. For one Prefill+Decode pair:

Prefill GPU: 1.5M tokens/sec, 96% cache hit → 1.44M hit tokens/sec (¥0.0288/sec) + 60K miss tokens/sec (¥0.06/sec).

Decode GPU: 14K output tokens/sec → ¥0.056/sec.

Pair revenue: ¥0.1448/sec → ¥0.0724/sec per GPU → ¥260.64/hour per GPU ≈ $38/hour.

Compared to market GPU rental/depreciation of $2–4/hour, gross margin exceeds 92% (assuming $3/hour operating cost). Notably, cache-hit tokens constitute >95% of throughput but only 19.9% of revenue; cache-miss inputs (4% of tokens) contribute 41.4% of revenue, and decode outputs (<1% of tokens) contribute 38.7%.

Traffic Pattern: Coding Agents Drive Long Contexts and High Cache Hits

The report highlights two key figures: average input length 77K tokens and average cache hit rate 96%. The author's own logs from using DeepSeek Harness (a coding agent) show similar patterns: 184K average context per request, 99.43% cache hit rate over 7,174 requests in three days. Coding agents (Claude Code, Codex, DeepSeek Harness) operate in loops: each step appends results to history and resends the entire context. Over 96% of each request is a repeated prefix, making KVCache reuse extremely effective.

Architecture: SSD Cache Cluster Offloads 96% of Prefill Compute

To avoid recomputing the 96% repeated prefix on GPUs, DeepSeek deploys a 3FS distributed SSD cluster (100 PiB ≈ 100,000 TB) alongside the 2,400 GPUs. The cluster delivers 10 TiB/s aggregate read bandwidth. When an agent sends the next request, the 96% historical context is read from 3FS via RDMA (GDR) directly into GPU memory, bypassing CPU and consuming zero GPU SM cycles. This software-hardware co-design amplifies effective Prefill throughput per GPU from 60K tokens/sec (recompute baseline) to 1.5M tokens/sec — a 25× gain. Cheap SSD capacity and network bandwidth replace expensive GPU compute, enabling the ¥0.02/M token cache-hit price.

SWA Bounded Replay: Dropping Sliding-Window Cache to Halve Prefill Compute

In the prior V4 design, the first 20 layers' sliding-window attention (SWA KVCache) was either stored on SSD or recomputed exactly (2,560 tokens) on cache miss. Production analysis showed SWA cache is only useful for minutes within an active dialogue but consumed nearly half the SSD capacity. Storing short-lived data on SSDs meant for day-scale retention was uneconomical.

V4.1 Flash introduces SWA Bounded Replay:

SWA cache is no longer written to SSD; kept only in producer machine's memory for 180 seconds (observed >98% in-memory hit rate).

On miss or at Decoder layers, instead of exact recomputation of 2,560 tokens, only the last 128 tokens are approximately recomputed.

This change frees ~50% SSD space and halves total Prefill compute.

Open-Source Components: DeepInfer and deepseek-recipe

DeepInfer packages single-node and super-node acceleration: FlashMLA, DeepGEMM, DeepEP, and DSpark variable-length speculative decoding (300 tokens/sec code generation). It also includes SWA Bounded Replay. The companion Rust library deepseek-recipe standardizes Chat Completions, OpenAI Responses, and Anthropic Messages protocol conversion, segmented parallel tokenization with injection safety, and cross-chunk streaming tool-call parsing.

For third-party cloud providers and enterprise teams, this provides an official-grade inference foundation without needing custom CUDA kernels or debugging streaming parsers. DeepInfer's README lists KVCache storage management, Prefill/Decode disaggregated scheduling, and cross-replica load balancing as not yet open-sourced due to internal infrastructure coupling, but the technical report fully exposes the cluster scheduling design and real revenue data.

Industry Impact: Validated Profitable Low-Cost Inference Path

The report demonstrates that with proper engineering (disaggregated Prefill/Decode, massive SSD KVCache pool, RDMA/GDR transfer, SWA Bounded Replay), large-scale inference can be highly profitable even at ¥0.02/M token cache-hit prices. This gives other vendors a concrete technical roadmap: adopt DeepInfer on alternative hardware (e.g., Huawei Ascend 950, also supported) to lower per-token hardware cost, and build distributed KVCache pools to replicate the SSD-for-GPU compute substitution. As more providers deploy similar architectures, API prices for coding agents and long-context workloads will continue to fall — a direct benefit for developers running agents daily.