DeepSeek open-sourced DeepInfer, the inference framework that powers its V4.1-Flash model, on day one of PyTorch Conf. The release includes the inference engine and API-processing layer, with support for distributed inference, speculative decoding, and structured output on NVIDIA and Huawei Ascend platforms. DeepSeek says future model inference implementations will also land in the repo.
According to the source's summary of DeepSeek's engineering report, one cluster serves 155 trillion tokens per day with a 96% input-cache hit rate, 400 ms average time to first token, and 230 tokens/s generation (300 tokens/s for coding). The report also claims $38 revenue per accelerator-hour at off-peak prices. Based on those numbers, the article estimates roughly $1M per hour in revenue and an inference margin above 80%, with an 81-day payback period. The source cautions that these are first-party production claims, not independently audited.
The article also notes a gap between "fully open source" and full reproducibility: DeepInfer's ultra-optimized KV cache and load balancing remain coupled to internal infrastructure and are not part of the release. Meanwhile, the vLLM team—whose engine DeepInfer was initially built on—published a matched-configuration benchmark showing decode throughput within 6% of DeepInfer's published GB200 results, qualifying the view that DeepSeek's serving efficiency is out of reach for existing open-source engines.