DeepInfer Reads Like an Answer to DeepSeek’s Own 2025 Objections

On October 20, 2026, DeepSeek announced DeepInfer, the inference framework it says serves DeepSeek-V4.1-Flash in production, and published the code under Apache 2.0. Eighteen months earlier, the same company had explained in writing why it would not release its engine. As The New Stack reported, the software was “impractical for public deployment without significant modifications.”

Both statements can be true at once, and the space between them is where this release gets interesting. DeepInfer is a substantial inference engine: kernels, a scheduler, a sampler, and multi-chip parallelism. The important question is where the public release stops and the surrounding production infrastructure begins.

This post reads the repository, its examples, and the first round of reporting to separate what you get from what you still have to build.

Quick take: DeepInfer is most useful as a reference implementation for inference engineers and as a path to running V4.1-Flash on supported hardware. DeepSeek’s speed claims have not yet been independently verified, and its documented batch-invariant and deterministic-inference features are NVIDIA only.

What DeepSeek said in 2025

DeepSeek’s earlier open-source push went component by component. In late February 2025, its Open Source Week shipped one building block a day: FlashMLA, an MLA decoding kernel for Hopper GPUs; DeepEP, an expert-parallel communication library; and DeepGEMM, an FP8 GEMM library. The week also covered the EPLB expert-parallel load balancer, the DualPipe training schedule, and the 3FS file system.

Inference code was the thin spot. For V3, the model card offered the DeepSeek-Infer demo, described as a “simple and lightweight demo” for FP8 and BF16, and pointed production users to community engines such as SGLang and LMDeploy.

In April 2025, the same index gained a post on open-sourcing the inference engine. DeepSeek explained that it had considered releasing the whole system and instead chose to contribute components to existing projects. It cited three obstacles: the engine had grown from an early vLLM version into a heavily customized system; a small research team would struggle to maintain another major open-source project; and the code was entangled with internal infrastructure.

The entanglement point matters most for what follows. The earlier post tied it specifically to cluster-management tooling.

DeepSeek also made a promise about hardware. It said it would synchronize inference engineering with partners ahead of future model launches, aiming for day-one availability across chip architectures. Keep that in mind for the Huawei section below.

What shipped on October 20

The repository supports one model, DeepSeek-V4.1-Flash, on two hardware families: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The default checkpoint is DeepSeek-V4.1-Flash-1017, the updated weights released days earlier. The README lists these features:

• Optimized kernels, model runtime, and sampler

• Attention DP and balanced CP, plus MoE expert parallelism with EPLB

• DSpark with confidence-scheduled verification

• SWA bounded replay

• Asynchronous scheduling, described as a zero-overhead batch scheduler

• Batch-invariant inference and deterministic sampling

• Structured outputs

Two companion facts shape the picture. DeepSeek credits vLLM as the base: DeepInfer was originally built primarily on it, with inspiration from SGLang and TensorRT-LLM. A second repository, deepseek-recipe, was updated alongside DeepInfer on October 20. Its Rust libraries and Python bindings handle request conversion, prompt encoding, and response parsing so self-hosters can match the behavior of DeepSeek’s official API. It does not replace the inference backend, tool execution, or HTTP transport.

The upstream loop DeepSeek described in 2025 is already visible. The Next Web reports that the vLLM project thanked DeepSeek and said it would keep improving its own V4.1-Flash support with ideas from DeepInfer. The repository had passed 8,000 stars by Tuesday afternoon, according to the same report.

timeline
    title DeepSeek path to DeepInfer
    Feb 2025 : Open Source Week : FlashMLA, DeepEP, DeepGEMM, EPLB, 3FS
    Apr 2025 : Post on open-sourcing the engine : Components go upstream, full engine stays internal
    Sep 2026 : DeepSeek-V4.1-Flash launches
    Oct 17 2026 : V4.1-Flash-1017 weights
    Oct 20 2026 : DeepInfer and deepseek-recipe updated

The scope is the argument

Here is the boundary, drawn from the README’s description of what a complete inference service still needs.

flowchart TB
    subgraph yours["Your serving stack"]
        LB["Load balancing across DP replicas"]
        DIS["Prefill/decode disaggregation orchestration"]
        KV["Long-lived KV-cache management and storage"]
    end
    subgraph shipped["Shipped in DeepInfer"]
        SCH["Asynchronous batch scheduler"]
        RT["Model and sampler runtime"]
        PAR["Attention DP, balanced CP, MoE EP with EPLB"]
        KER["FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, TileLang kernels"]
    end
    LB --> DIS
    DIS --> SCH
    KV --> SCH
    SCH --> RT
    RT --> PAR
    PAR --> KER

The announcement called DeepInfer “fully open source.” The README is more specific: it describes offline inference with core features needed for serving at scale, then lists three components not included in the public release: a KV-cache system, prefill/decode disaggregation (while noting that most of the required engine functionality is already implemented), and load balancing across data-parallel replicas for both prefill and decode.

One distinction matters here: the missing KV-cache system does not mean the engine cannot use KV state during ordinary inference. The gap is the broader cache-management and storage system needed for long-lived prefix reuse and production-scale routing.

DeepSeek’s deployment Q&A makes the infrastructure boundary explicit. It says the company’s KV-cache management and load-balancing components remain closed because they are tightly coupled to internal infrastructure. The document also describes the storage and routing architecture behind those components.

That helps explain why the public release stops short of a complete serving system. It does not, however, answer every question raised in 2025. DeepSeek previously cited limited maintenance capacity and a heavily customized vLLM fork. The new release acknowledges its vLLM roots, but the company has not fully explained what changed to make a separate, model-specific engine sustainable as an open-source project.

For planning, the line is simple. Offline batch inference, evaluations, and research on a supported cluster: DeepInfer is close to complete. A multi-tenant chat API with long-lived prefix caching, replica routing, and split prefill/decode pools: budget real engineering time for the layer above it.

Inside the engine

The feature list is easier to judge with the switches next to it.

Read the last column twice. The DSpark recipe sets DEEPINFER_ENABLE_MTP_VARLEN=1, while the deterministic-sampling recipe sets it to 0. If you need reproducible outputs for evaluations or regression tests, the two documented recipes pull in opposite directions. The README does not quantify the performance trade-off, so test before assuming they compose.

Bounded replay exposes a model-specific constant. The README says the retained suffix must cover the sliding-window-attention window plus the maximum backtracking length, and gives 140 tokens as the default for V4.1. Treat 140 as a V4.1 setting, not a universal one. Anyone adapting the engine to another model has a value to derive rather than copy.

The kernel list brings the earlier component releases together with TileKernels, DeepSelect, and custom TileLang kernels. DeepInfer is where those specialized components meet a scheduler and a model runtime.

Reading the benchmark charts without fooling yourself

The README shows two Pareto plots: prefill (non-cached input throughput against latency) and decode (output throughput against per-user speed). DeepSeek measured them on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and points to run_sweep.sh for reproduction. The Next Web relays Bloomberg’s note that no one has independently benchmarked DeepInfer’s gains yet.

The examples README documents the harness in unusual detail. One line changes how you should read the decode curve. The harness reports throughput using full-acceptance accounting:

decode tokens/s/GPU = avg_SPS x decode_num_requests x (1 + nextn) / cp

With 64 requests and a draft length of 5, this credits 384 tokens per step before dividing by context-parallel size. That is the full-acceptance case, not the configuration shown in the published decode chart.

The published chart is labelled nextn=3 fixed, with 75% per-token acceptance and (\tau=3.25). The plotting script applies that acceptance assumption to both per-user speed and GPU throughput. In other words, the plotted figures are adjusted using an assumed acceptance rate rather than simply showing the raw full-acceptance ceiling.

The 75% assumption is explicit, but it is still an assumption. Before comparing DeepInfer against vLLM or SGLang, check the acceptance rate under your own workload and ensure both sides count generated tokens the same way.

The methodology has strengths worth copying. The harness separates prefill from decode, drops intervals that collide with MoE load-balance statistics flushes, and runs sweeps with the profiler off because it perturbs timings. It also warns that profiler-traced step times can be inflated by nearly 50% at batch size 1 and must not be used to derive steps per second.

Numerics are held to an exact standard. Perplexity baselines live in tests/models/offline_decode_ppl.json, keyed by platform and workload, and validation compares them sample by sample. A refresh_offline_decode_ppl.py script updates them once a numeric change is confirmed as expected, and the docs say to re-measure on NVIDIA and Ascend separately. A --check_structured_output flag extends the same run to regex, JSON, and EBNF checks.

NVIDIA and Huawei, side by side

The hardware list is the most strategic line in the README. Next to NVIDIA’s SM100 parts, which The New Stack identifies as Blackwell data-center GPUs, sits Huawei’s Ascend 950 series, with CANN 9.2 as its prerequisite. Older NVIDIA architectures such as Hopper are absent from the supported list.

This sits inside a broader run of reporting. Bloomberg reported in September that DeepSeek planned to deploy at least 160,000 Ascend 950DT chips at a new Inner Mongolia data center. Reuters reported later that month that DeepSeek and Huawei were collaborating on tools to make Ascend chips easier to program. The Next Web adds that DeepSeek published tooling for Huawei chips earlier in October as an alternative to CUDA.

Parity is incomplete: the published performance charts use NVIDIA GPUs, and the README does not establish how DeepInfer performs on Ascend under comparable workloads.

The examples do show Ascend treated as a real target. They document debugging variables such as ASCEND_LAUNCH_BLOCKING=1, which runs operators synchronously so stack traces are easier to read, and they keep separate numeric baselines per platform.

This matches the 2025 promise of day-one cross-chip support, delivered for one model on two chip families. It is a narrow delivery of a broad commitment. The Next Web reports that DeepSeek says future models and improvements will land in the same repository.

Running it today

The prerequisites are Linux, GCC 13 or newer, Python 3.12, and either CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). Start from an empty Python 3.12 environment.

export DEEPINFER_ACCELERATOR=nvidia   # use "ascend" on Ascend
./build.sh

# first run downloads DeepSeek-V4.1-Flash-1017 into the Hugging Face cache
python examples/offline_inference.py --ep 8

The feature flags and their implications are listed in the table above; note that the documented DSpark and deterministic-sampling recipes set DEEPINFER_ENABLE_MTP_VARLEN differently.

Parallel scale comes from the environment, not the command line. In the benchmark harness, ep_size equals WORLD_SIZE times the number of visible devices, and --cp must divide it. Redundant experts default to 32 through DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS, but the engine rounds the total up so the expert count divides evenly by ep_size. Set DEEPINFER_HF_MODEL_DIR to run from a local copy of the weights.

There is also a difference between running the offline example and operating a production service. DeepSeek’s deployment Q&A describes a much larger split deployment: typically 64–128 GPUs for the prefill pool and another 64–128 for decode, with its smallest efficient online configuration described as 64 GPUs for prefill plus 64 for decode. These are DeepSeek’s production recommendations, not minimum requirements for running the offline example.

Who should adopt it

What to watch next

Four signals will show how far DeepInfer travels beyond the announcement.

• Independent benchmarks on identical hardware and identical token accounting, against vLLM and SGLang.

• The missing layers. DeepSeek’s deployment Q&A confirms that its KV-cache management and load-balancing components remain closed. Watch for any change in that boundary.

• What lands upstream. vLLM said it would use ideas from DeepInfer. Watch for merged changes rather than statements.

• Ascend parity for batch-invariant inference, deterministic sampling, and independently measured performance.

The practical test now is whether outside teams can reproduce the published results and whether ideas from the code make their way into upstream engines. Until then, DeepInfer is most compelling as a reference implementation and a pilot for teams already working on its supported hardware.