DeepSeek open-sources DeepInfer, its large-scale AI inference framework, with support for NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs

DeepSeek has released DeepInfer, the framework behind its production inference services, as open source under Apache 2.0. The release supports DeepSeek-V4.1-Flash and runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, requiring Linux, GCC 13+, Python 3.12, and CUDA 13 or CANN 9.2. Features include optimized kernels, MoE expert parallelism with EPLB, speculative decoding via DSpark with confidence-scheduled verification, batch-invariant inference, deterministic sampling, and structured outputs. Benchmarks used 4 SM100 GPUs for prefill and 64 for decode. KV caching, disaggregation, and load balancing are not included. The project builds on vLLM, SGLang, and TensorRT-LLM.

✨ AI summary of the linked source — not editorial content.

Explore the topic

• AI