Evidence at a glance
An Inference Framework, Not a Complete Production Service
DeepSeek has released DeepInfer, the framework behind online inference for its V4.1-Flash model, targeting throughput, latency, and scheduling challenges in large-scale model serving. The repository also includes API-processing components and supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output on NVIDIA and Huawei Ascend platforms. DeepSeek says future model inference implementations and optimizations will also be added to the repository.
But an open inference framework is not the same as an open production service. The repository says DeepInfer is built primarily on vLLM and acknowledges ideas from projects including SGLang and TensorRT-LLM. More importantly, the KV-cache management, stage separation, and load balancing used by the online service have not been fully released. What is available is not a turnkey blueprint spanning hardware, serving, and traffic orchestration, but an inspectable engine alongside system boundaries operators must still fill in.
Inference Efficiency Shifts from Computation to State Reuse
To understand DeepInfer’s efficiency strategy, start by separating the work involved in generation. Vision encoding and prefill process the input, while decode produces output token by token. These stages have different compute and resource profiles, so DeepSeek’s production design deploys them separately and schedules them according to their respective loads. The goal is not to push one GPU to peak utilization across every stage, but to reduce contention and let each stage scale independently.
An even more important variable is the KV cache, the model state accumulated while processing context. When a long conversation or repeated prefix can reuse that state, the system avoids recomputing it from scratch. DeepSeek says its production design stores persistent KV in shared storage, so a request need not return to the node that first handled it. For short-lived SWA state, SWA Bounded Replay reconstructs state by replaying only the latest 128 tokens; the engineering report says this brought total prefill computation close to half. The optimization target is therefore not just tokens generated per second, but also which computations can be skipped and how cached state survives across requests.
155 Trillion Tokens a Day Is First a Cache Statistic
DeepSeek’s reported production figures are striking: one cluster serves 155 trillion tokens per day, input-cache hits reach 96%, average time to first token is 400 milliseconds, and generation runs at 230 tokens per second, rising to 300 for coding tasks. The report also breaks out rates by GPU: each prefill GPU processes about 1.5 million cache-inclusive input tokens per second, while each decode GPU generates about 14,000 tokens per second. These figures describe different parts of the pipeline; they should not be collapsed into a single throughput number for arbitrary workloads.
The 96% cache hit rate is particularly important. It means the 155-trillion total reflects, to a large extent, the system’s ability to reuse repeated inputs and existing state, not the full computation of an equivalent volume of entirely new prompts. DeepSeek also reports about 100 PiB of SSD for persistent KV storage, with different retention periods by traffic type: three days for agent traffic and seven days for chatbots. Such mechanisms can change costs substantially for services with repeated context. For workloads with little cache reuse and highly varied prompts, the headline figure is not a direct basis for capacity planning.
Margins and Payback Are Estimates, Not Audited Results
Discussion of DeepInfer has also translated throughput into business economics. Using off-peak prices, an estimated all-in cost of roughly $5 to $8 per accelerator-hour, and DeepSeek’s reported revenue figures, the supplied material estimates inference margins above 80% and a payback period of about 81 days. Revenue per unit of compute and payback speed matter when a model provider decides whether expansion is worthwhile. But these are estimates built on public prices and cost assumptions, not audited operating results.
The calculation is highly sensitive to cache performance and utilization. A lower cache hit rate means more input must be computed; different models, hardware, traffic patterns, and pricing arrangements also change revenue and accelerator-hour costs. The accompanying cluster and per-gigawatt extrapolations are best read as scenarios, not returns that another organization can expect simply by deploying DeepInfer. Evaluation teams should measure throughput, cost, and cache-hit rate under the same workload conditions rather than citing a margin or payback period in isolation.
When Engine Performance Converges, the System Boundary Matters
The vLLM team says that, in a matched GB200 comparison, its decode throughput was within 6% of the result reported for DeepInfer. That is not an independent evaluation proving the two systems equivalent across all workloads. It does, however, point to a shift worth noting: when engine-level performance is close, an advantage may come from caching, stage orchestration, load balancing, and data-center operating conditions rather than from the inference framework’s name alone.
That is why open code and reproducible production performance are different things. Users can evaluate DeepInfer’s released speculative decoding, structured output, and replay mechanisms, but must still design the cache and scheduling layers that have not been fully released and determine how they fit existing infrastructure. For workloads dominated by long conversations, repeated prefixes, or agent calls, shared KV and state retention may deserve early testing. If cache hits are uncommon, or the storage and orchestration capabilities are missing, the reported throughput and cost advantages may not transfer. Start with the cache-hit rate and end-to-end cost of your own traffic, then decide whether to migrate; do not start with a cluster record.