DeepSeek Open-Sources Its Inference Engine, Not the Full Service

An Open-Sourced Layer of the Production Engine

DeepSeek has open-sourced DeepInfer, the inference framework behind its DeepSeek-V4.1-Flash production service. Released under Apache 2.0, the current package targets offline inference. It brings together model runtime, sampling, inference kernels, and parallel scheduling to execute a model efficiently on particular hardware; it is not an out-of-the-box online serving platform.

That distinction matters when reading the phrase “production inference framework.” DeepSeek says it uses DeepInfer in large-scale production, but the release does not include the KV-cache system, disaggregated Prefill and Decode deployment, or load balancing across data-parallel replicas. What is public is an important execution layer inside the online system, not the complete path from incoming request through routing and queueing to generated output.

Performance Comes from Coordination, Not One Kernel

DeepInfer is built on components including FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels, with model and sampling runtimes above them. DeepSeek says the framework was initially built primarily on vLLM and also drew ideas from SGLang and TensorRT-LLM. Its engineering contribution is therefore not just a collection of operators, but a view of how kernels connect to a runtime and work alongside parallelism and scheduling.

For parallel execution, the attention portion uses data parallelism while the MoE portion uses expert parallelism, with an expert-parallel load balancer handling assignment on the expert side. DeepSeek describes production configurations that vary by stage: Decode can use EP64 to EP128 within a supernode, while Prefill typically uses the smallest expert-parallel configuration that fits in memory, around EP2 to EP8. The published tests used four SM100 GPUs for Prefill and 64 for Decode. Those are benchmark conditions, not mandatory deployment sizes.

Two Optimizations Aim to Avoid Wasted Work

DeepInfer's DSpark speculative decoding does more than ask a draft model to produce extra tokens. The draft proposes five tokens per step, a Confidence Head estimates the chance each one will be accepted, and a scheduler chooses how much of the prefix to verify. The aim is to spend verification compute on candidates more likely to be useful instead of treating every draft token alike.

DeepSeek reports that dynamic verification reduces the average number of tokens verified from five to roughly 3.2–4.4, while average accepted length changes only slightly, from 3.29 to 3.20. At a test point of about 60 sequences per rank, DeepSeek reports a throughput gain of roughly 4%–7%. The figures suggest that avoiding verification work can translate into a benefit, but the gain depends on workload conditions and should not be treated as a universal speedup for every batch size or model configuration.

A second way to avoid repeated work is SWA bounded replay. On a cache hit, the system recomputes only a recent suffix to restore sliding-window attention state instead of recalculating the full context. The deployment Q&A describes a 128-token suffix, while the README gives a default retained suffix of 140 tokens. DeepSeek says this keeps replay cost from growing with context length and substantially reduces Prefill computation in the relevant layers. Those two values come from different descriptions and should be read as a difference in configuration or documentation, not as a single fixed constant.

The Throughput Claims Have Specific Test Conditions

The available performance figures come mainly from curves published by DeepSeek and read from those charts by the report; they are not independent third-party benchmarks. At a context length of 131,072 with EP64 and other stated conditions, the Decode curve exceeds 25,000 output tokens per GPU per second at 256 requests per rank, with per-user speed near 100 tokens per second. At eight requests per rank, per-user speed rises above 300 tokens per second, illustrating the different trade-offs between aggregate throughput and responsiveness for an individual user.

The Prefill curve uses four SM100 GPUs. At 4,096 input tokens per rank, non-cached throughput is about 60,000 tokens per GPU per second. At 65,536 input tokens, it approaches 90,000, while step latency is roughly 730 milliseconds. These figures cannot be compared meaningfully without the input length, request batch, parallel configuration, and hardware model. DeepSeek also says one cluster serves 155 trillion tokens per day, but that is the company's account of its production scale, not independently verified performance or cost data.

Before Reusing the Framework, Account for the Missing Service Layer

For technical leaders, DeepInfer's clearest value is that it makes performance questions legible as engineering choices: how kernels are composed, how parallelism is allocated between Prefill and Decode, how speculative verification is bounded, and how CPU scheduling overlaps asynchronously with GPU execution. The framework also supports structured output constrained by regular expressions, JSON Schema, and EBNF. Batch-invariant inference and deterministic sampling are listed as NVIDIA-only, a reminder that capabilities should not be assumed to behave identically across hardware.

Using it for online service still means building what the release leaves out, including KV-cache management, stage disaggregation, request routing, and load balancing. Public support is also concentrated on DeepSeek-V4.1-Flash and NVIDIA SM100 and Huawei Ascend 950 hardware, so the release does not establish performance on other models or devices. Teams should first reproduce end-to-end metrics with their own context lengths, concurrency patterns, and hardware, then decide whether kernel-level gains justify the additional systems work. DeepInfer opens a window onto the inference execution layer; the service complexity beyond that window remains the team's responsibility.