vLLM Adds Day-0 Kimi K3 Support: What It Means for Local Serving

The new models grabbed the headlines this month, but the quieter and arguably bigger story is that the open-source runtimes finally caught up to them. On July 16, Moonshot AI announced Kimi K3 - a 2.8-trillion-parameter mixture-of-experts model - and in the same breath revealed that its own engineers had contributed a new KDA (Kimi Delta Attention) prefix-caching implementation directly upstream to vLLM, so the community serving stack would support the model on day zero when open weights land on July 27. That upstream-first pattern, repeated across vLLM, Ollama and llama.cpp all through July, is what actually decides whether trillion-parameter open weights are usable on your own hardware or just a press release.

For most of the open-weight era, the gap between a model announcement and usable open-source inference was measured in weeks. A lab would publish weights with a custom attention trick or an exotic quantization layout, and the vLLM and llama.cpp maintainers would spend the next fortnight reverse-engineering it before anyone outside a datacenter could serve it efficiently.

What changed in July is where the work happens. The Kimi team did not wait for maintainers to figure out K3's attention; they upstreamed the KDA prefix-caching code into vLLM themselves, timed to release alongside the weights. KDA deliberately breaks the assumptions behind conventional prefix caching, so without that contribution long-context serving would have been slow for weeks. Instead, the day the weights drop, the fast path already exists in the mainline runtime.

This matters for anyone running AI locally because the runtime, not the model card, is what determines your real-world throughput and memory footprint. If you have followed our [Kimi K3 run guide](/blog/kimi-k3-largest-open-weight-model-run-locally), you already know the raw weights are enormous - the interesting question is which runtime lets you serve them without paying for eight extra GPUs.

The trick is sparsity. These are mixture-of-experts models, so the total parameter count is not what you compute per token. Kimi K3 activates only 16 of its 896 experts at a time; DeepSeek V4 Pro carries 1.6 trillion total parameters but activates roughly 49 billion; GLM-5.2 is 744 billion total with about 40 billion active; and Thinking Machines' Inkling is 975 billion total with about 41 billion active. Runtimes exploit that gap so the FLOPs per token stay comparable to a dense mid-sized model even as the weights balloon.

The other half of the problem is the KV cache, which grows with context length and quietly dominates memory at long contexts. vLLM's DeepSeek V4 backend is a good example of how much runtime engineering hides here: it reports roughly an 8.7x smaller KV cache than V3.2-style models, needing only about 9.62 GiB per sequence at a full 1-million-token context in bf16, and that halves again with fp4 or fp8. If you want the mechanics behind loading, memory and quantization, our explainer on [how local LLM inference actually works](/blog/how-local-llm-inference-actually-works) walks through it step by step.

Sources: Moonshot AI (Kimi K3), DeepSeek model cards, Z.ai (GLM-5.2), Thinking Machines (Inkling), cross-checked against MarkTechPost's July 18 comparison. Note that active parameters per token are far smaller - 40 to 49 billion for the disclosed models - which is why runtimes can serve them at reasonable speed.

vLLM did the heaviest lifting. It has supported the DeepSeek V4 family since late April - both the 1.6T DeepSeek-V4-Pro and the 285B DeepSeek-V4-Flash - targeting NVIDIA Hopper and Blackwell, with reference deployments of 8x B200/B300 for Pro and 4x B200/B300 for Flash. The Kimi K3 KDA contribution now extends that day-0 posture to Moonshot's model, meaning the community gets efficient long-context serving from the moment weights are public.

Ollama shipped at a startling cadence, moving from v0.31.1 to v0.32.2 in three weeks. The July 1 release delivered nearly 90% faster token generation for Gemma 4 on Apple Silicon via multi-token prediction; July 8 turned on flash attention for older NVIDIA cards (compute capability 6.x), which directly helps people on aging hardware; and the mid-to-late July releases added a new interactive agent experience and a skills system. Multi-token prediction is the same speculative-decoding family that has been quietly doubling single-user throughput across runtimes this year.

llama.cpp, meanwhile, kept its continuous-build train rolling from roughly b10066 to b10075 in mid-July, and recent work added MXFP4_MOE quantization names plus MoE prefill optimizations - exactly the plumbing you need to run giant sparse models with experts offloaded to CPU RAM. If you are weighing which quant to actually download, our breakdown of [Q4 vs Q5 vs Q6 vs Q8 quality loss](/blog/q4-k-m-vs-q5-k-m-vs-q6-k-vs-q8-0-quantization-quality-loss) is the companion piece.

# vLLM: serve the genuinely-servable member of the family, DeepSeek-V4-Flash (285B).
# fp8 KV cache keeps the 1M-token context affordable; adjust TP size to your GPUs.
pip install -U vllm
vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --tensor-parallel-size 4 \
  --max-model-len 1000000 \
  --kv-cache-dtype fp8

# Ollama: pull an MoE that fits a workstation, with July's faster runtime built in.
ollama pull glm-5.2
ollama run glm-5.2

# llama.cpp: run a giant MoE by keeping experts on CPU RAM and only offloading
# attention/dense layers to the GPU (-ot overrides tensor placement).
llama-cli -m ./glm-5.2-Q4_K_M-00001-of-000NN.gguf \
  -ngl 99 -c 8192 \
  -ot "\.ffn_.*_exps\.=CPU" \
  -p "Write a Python function to parse a GGUF header."

The honest answer for the flagships is no. Kimi K3 recommends 64-plus accelerators, GLM-5.2 self-hosting wants north of a terabyte of VRAM at full precision, and DeepSeek V4 Pro's 1.6 trillion parameters put single-node serving out of reach for most teams. These are datacenter models, and the vLLM deployment examples - eight B200s here, four there - make that explicit.

But the runtime work trickles down, and that is the point of tracking it. The same MXFP4 and MoE-offload code that lets a datacenter serve K3 is what lets you run a quantized GLM-5.2 or DeepSeek-V4-Flash on a workstation with experts streamed from CPU RAM. Flash attention for compute-capability 6.x cards means a five-year-old GPU gets faster. If you are trying to figure out which of these giants your rig can realistically touch, our head-to-head on [which giant MoE fits your hardware](/blog/glm-5-2-vs-deepseek-v4-vs-kimi-k3-hardware-fit) maps the tiers.

A few more releases worth your attention, each with the primary source:

• ▸Thinking Machines Lab - Mira Murati's startup - shipped its first model, Inkling, on July 15 as an open-weight MoE with 975B total and about 41B active parameters, a rare open release from a US frontier lab that reportedly uses a third the tokens of Nvidia's Nemotron 3 Ultra for the same coding score. (Source: TechCrunch, techcrunch.com/2026/07/15)
• ▸Ollama reached v0.32.2 on July 20, adding a skills system for agents and unlimited tool rounds for cloud models by default, on top of the July 1 update that made Gemma 4 nearly 90% faster on Apple Silicon. (Source: Ollama release notes via Releasebot, releasebot.io/updates/ollama)
• ▸llama.cpp crossed build b10075 in mid-July, adding MXFP4_MOE quant support and MoE prefill optimizations that make offloading experts to CPU RAM practical for consumer machines. (Source: ggml-org/llama.cpp releases, github.com/ggml-org/llama.cpp/releases)
• ▸vLLM confirmed that Moonshot's team upstreamed a KDA prefix-caching implementation for day-0 Kimi K3 support when weights release on July 27, extending the day-0 serving it already offers for DeepSeek V4-Pro and V4-Flash. (Source: vLLM on X, x.com/vllm_project)

Runtime headlines are exciting, but the practical question is always the same: does it fit? Before you download a 200 GB GGUF, run the numbers. The [Runyard VRAM Calculator](/tools/vram-calculator) tells you which quant of GLM-5.2, DeepSeek-V4-Flash or Qwen3.6 fits your exact VRAM, how much context you can hold, and the tokens-per-second you can expect - including CPU-RAM expert offload scenarios like the llama.cpp command above.

If you are choosing between models rather than quants, Runyard Compare puts them head to head on hardware fit and cost, and Model Radar on the homepage ranks what will actually run on the GPU you own. The runtimes did their job this month; these tools tell you what that unlocks for your machine.