DeepSeek gives away the engine behind its cheap AI

DeepSeek just gave away the thing that made it the most efficient AI company on the planet.

On October 20, 2026, the Hangzhou-based lab open-sourced DeepInfer, the production inference framework that powers its entire API. This isn't a research prototype or a stripped-down demo — it's the same code that handles 155 trillion tokens per day across a single cluster. Under the Apache 2.0 license, anyone can inspect it, modify it, and use it commercially.

The announcement landed on X at approximately 14:08 (UTC) and immediately pushed the repository to the #1 spot on GitHub Trending. Alongside DeepInfer, DeepSeek also released deepseek-recipe, a production-grade token encoding/decoding component that mirrors the exact behavior of its official API, under the MIT license.

What DeepInfer actually does

If model weights are the brain, inference software is the circulatory system — it determines how fast, how cheaply, and at what scale the brain can actually think. DeepSeek's prices have been a fraction of OpenAI's and Anthropic's for over a year, and the industry has long wondered exactly how they pull it off. Now we know.

The framework is built on a vLLM base with ideas borrowed from SGLang and TensorRT-LLM, but layered with DeepSeek's own custom kernel libraries — FlashMLA, DeepGEMM, DeepEP, TileKernels, and DeepSelect — plus custom TileLang kernels. On top of that foundation sit several techniques that are genuinely distinctive:

• DSpark speculative decoding with confidence-scheduled verification: instead of always verifying a fixed number of draft tokens, the engine estimates each draft's acceptance probability and skips verification for low-confidence ones, routing that compute to other requests. DeepSeek reports a 6% throughput gain over fixed 5-token verification.
• SWA bounded replay: when a cache hit occurs, the model recomputes only the last 128 tokens of sliding-window attention instead of the full 2,560. This cuts prefill computation by roughly half for the 20 layers that use SWA.
• MoE expert parallelism with EPLB (Expert Parallel Load Balancing): V4.1 Flash has 384 routing experts. During decode, DeepInfer spreads 6 experts per GPU plus 1 replica of the hottest expert, rebalancing every 1,000 steps and redistributing every 10,000. The busiest card runs only 9% hotter than average.
• Asynchronous batch scheduling enabled by default: the CPU stages the next batch while the GPU is still computing the current one, eliminating idle time between steps.

The numbers behind the curtain

DeepSeek's deployment Q&A — published alongside the code — reveals production metrics that most labs guard as trade secrets:

The 25x gap between cached and uncached prefill is the headline number. In agentic workflows — where every tool call re-sends the system prompt, tool definitions, and entire conversation history — 96% of that work is redundant. DeepSeek stores KV caches in a dedicated 100-petabyte SSD cluster (their own 3FS filesystem), keeping agent caches for 3 days and chat caches for 7 days. Any prefill machine can read any cache segment, so users aren't pinned to a specific server.

The minimum production deployment is 64 GPUs for prefill plus 64 for decode — the two stages are disaggregated onto different hardware because they have fundamentally different resource profiles. Prefill is compute-bound (processing tens of thousands of tokens at once), while decode is memory-bound (generating one token at a time, reading parameters and KV cache from VRAM).

Why Huawei Ascend support matters

DeepInfer supports NVIDIA SM100 (Blackwell-generation, CUDA 13) and Huawei Ascend 950 (CANN 9.2) from day one. This isn't a footnote — it's arguably the most strategically significant detail in the entire release.

Export controls have restricted China's access to the newest NVIDIA chips. By shipping first-class Ascend support in its production inference engine, DeepSeek is demonstrating that a frontier AI lab can run at global scale on domestic silicon. For Chinese enterprises and government buyers evaluating hardware diversification, this is concrete proof of concept. For international buyers nervous about single-vendor dependence on NVIDIA, it offers a credible alternative path.

What's missing — and why that matters

DeepSeek has been upfront about the gaps. The open-source release provides offline inference only. Three critical components of the full production service are not included:

1. KV cache management — the 3FS storage layer and cache routing logic
2. Prefill/decode disaggregation — the machinery to split stages across separate machine pools
3. Replica load balancing — distributing requests across data-parallel replicas

The design details for all three are documented in the deployment Q&A, but the code itself remains internal. This means DeepInfer today is a high-performance serving core, not a drop-in replacement for a managed inference cloud. A team wanting production-grade self-hosting will need to build or bolt on their own caching layer, load balancer, and disaggregation logic.

That limitation is worth stating plainly because it tempers the "DeepSeek just gave away everything" narrative. What they gave away is the engine block — the optimized kernels, the scheduler, the speculative decoding, the MoE parallelism. The transmission, the fuel tank, and the steering wheel are still in Hangzhou.

Why it matters

The inference layer has always been the real moat in AI serving, and until now every frontier lab has treated it as proprietary. OpenAI and Anthropic have never released their production serving code. vLLM and SGLang are excellent community projects, but they're built by outsiders reverse-engineering optimal serving strategies. DeepInfer is the first time a top-tier lab has said: here is exactly how we do it.

This changes the competitive dynamics in three ways.

For the open-source community, it closes the gap between "having model weights" and "running them efficiently." Plenty of organizations downloaded DeepSeek's open weights only to discover that serving them at acceptable latency required infrastructure expertise most teams don't have. DeepInfer gives those teams a company-authored, production-proven path.

For competitors, the pressure is now on inference efficiency as a visible, comparable metric. When DeepSeek's serving code is public, every other lab's internal serving team will be asked why they can't match 14,000 tokens/sec/GPU or 96% cache hit rates. The era of treating inference optimization as a hidden competitive advantage may be ending.

For the US-China technology relationship, the Ascend support sends a message that transcends AI: Chinese frontier AI is not dependent on NVIDIA silicon to operate at scale. Whether that accelerates export controls or prompts a rethinking of them is a political question, but the technical proof point is now public.

DeepSeek researcher Deli Chen framed the release in starkly ideological terms on the announcement thread: "Open source versus closed source. Light versus darkness. A decisive battle between two futures, two destinies of humanity. We will fight as long as they want. Until complete victory." Whether you find that stirring or overheated, the code behind the rhetoric is now available for anyone to audit.

A critical look

A few caveats deserve mention. The $38 revenue-per-GPU-hour figure is calculated at "valley" electricity prices — off-peak rates that may not be available in every market. The 155-trillion-token-per-day number is per cluster, and DeepSeek hasn't disclosed how many clusters it operates, so total throughput remains unknown. And the framework currently supports exactly one model — V4.1 Flash — meaning its utility is bounded until DeepSeek adds support for future releases.

There's also a strategic reading that cuts against the "pure altruism" framing. DeepSeek may be open-sourcing DeepInfer precisely because its inference advantage is eroding — vLLM and SGLang have been catching up rapidly, and the marginal value of keeping the engine proprietary may have declined below the value of ecosystem lock-in. If every startup and cloud provider builds on DeepInfer, DeepSeek's model format and tokenization become de facto standards, creating switching costs that benefit DeepSeek far more than any short-term inference edge.

What to watch next

• DeepInfer updates: KV cache management and replica load balancing are the obvious next components to land. If they appear within 3 months, DeepSeek is serious about making this a full production stack.
• Competitor responses: Watch for Qwen, Kimi, or Zhipu to open-source their own inference frameworks. DeepSeek has repeatedly set templates that Chinese competitors match.
• Enterprise packaging: Infrastructure startups will likely emerge offering managed DeepInfer deployments, mirroring the commercial ecosystem around vLLM.
• Ascend adoption metrics: If real-world benchmarks show Ascend 950 running DeepInfer within 10-15% of SM100 performance, the hardware diversification narrative becomes much harder to dismiss.
• Model support cadence: DeepSeek has promised future model inference code in the same repo. The gap between a new model announcement and DeepInfer support will tell us whether this is a genuine commitment or a one-time publicity move.

The era of open model weights is well established. DeepInfer may mark the beginning of the next phase: open serving infrastructure. Whether that benefits DeepSeek or the broader ecosystem more is the question the next 12 months will answer.

Sources: DeepSeek GitHub (deepseek-ai/DeepInfer), DeepSeek deployment Q&A, AI/TLDR DeepInfer coverage, 36Kr/新智元, SHAttered.io, The Next Web, SiliconANGLE, The New Stack.