Reportedly: DeepSeek open-sources DeepInfer, the inference engine behind DeepSeek-V4.1-Flash
Model: DeepSeek-V4.1-Flash · availability, license and releases
Reported by Leiphone · not yet confirmed by the company or a second independent outlet. We update this page when it is.
DeepSeek open-sourced DeepInfer on GitHub, the inference engine behind DeepSeek-V4.1-Flash, including MoE expert parallelism, asynchronous batch scheduling, custom kernels, and speculative decoding. The company disclosed that a single cluster processes about 155 trillion tokens daily. Currently only offline inference is open; online KV cache storage and cross-replica load balancing are not yet fully public, Leiphone reported.
China context
- Original name
- 深度求索
- Outside China
- Open weights · leiphone.com
- Claims
- Company-reported; not yet independently evaluated
- For builders
- Builders can study DeepInfer's asynchronous scheduling and MoE parallelism code to optimize their own inference serving.
- For investors
- Investors can monitor whether DeepInfer's open-sourcing accelerates adoption of DeepSeek's models or pressures competing inference providers.
Translated from Chinese. Quotes and facts link to the original sources.
DeepSeek released DeepInfer, the inference engine powering DeepSeek-V4.1-Flash, on GitHub. The open-sourced components include MoE expert parallelism, an asynchronous batch scheduler called Zero-Overhead Batch Scheduler, custom kernels, and speculative decoding. DeepSeek reported that a single cluster handles approximately 155 trillion tokens per day. The release currently covers offline inference; online KV cache storage and cross-replica load balancing remain closed. The Zero-Overhead Batch Scheduler overlaps CPU scheduling with GPU forward passes to reduce idle time during decode, addressing state management across rounds.
DeepInfer's Zero-Overhead Batch Scheduler lets the CPU schedule the next computation one round ahead, overlapping scheduling with model forward execution. This requires managing sequence length, KV cache state, and request lifecycle across rounds where the previous round's tokens may not have completed CPU post-processing. The open-sourced code shows how DeepSeek organizes computation, communication, and request scheduling for large-scale inference.
Developers outside China can now inspect and potentially reuse DeepSeek's inference optimizations for MoE models, reducing the cost of serving large models. Competitors can benchmark their own inference engines against DeepInfer's disclosed techniques.
Open-sourcing DeepInfer may lower inference costs for organizations deploying MoE models and could attract contributions or adoption from the developer community.
The next verifiable signal is whether DeepSeek releases the online KV cache storage and cross-replica load balancing components, or whether third-party benchmarks confirm DeepInfer's throughput and latency claims.