How to Run DeepSeek-V4-Flash Locally: 284B MoE at ~36 tok/s on 192GB VRAM

DeepSeek-V4-Flash is the model every local-AI channel is quantizing this month, and the reason is the ratio: 284 billion total parameters but only 13 billion active per token, wrapped around a 1-million-token context window. Unsloth shipped dynamic GGUFs for it in the 2026.7.2 release, and a working llama.cpp branch already runs it at roughly 36 tokens per second on two Blackwell cards. The catch is size - even the smallest usable quant is 103 GB. This guide maps every quant to real memory, shows the measured speed, and gives you the exact commands to load it.

DeepSeek released V4-Flash on April 24, 2026 under an MIT license, and it is the smaller, faster sibling in the V4 family. Where the full DeepSeek-V4 is a 1.6-trillion-parameter system, Flash trims the design to 284 billion total parameters while routing only about 13 billion of them per token. Its router picks the top 6 of 256 routed experts, plus one always-on shared expert, so every token pays for a fraction of the network.

The architecture is built for long context. Flash combines two attention schemes - Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) - to keep the KV cache manageable across its 1-million-token window, with a 384K maximum output length. On paper it scores 86.2% on MMLU-Pro, 88.1% on GPQA Diamond, and 79.0% on SWE-bench Verified, which puts it in the same tier as much larger models on reasoning and coding.

It became the model of the week for a practical reason: Unsloth published dynamic GGUF quants and DeepSeek-V4-Flash support with thinking toggles in the 2026.7.2 release, and a llama.cpp work-in-progress branch made local inference possible outside of a datacenter. When Unsloth quantizes something, the local-run crowd follows within days.

• ▸Total parameters: 284B, with about 13B active per token (top-6 of 256 routed experts plus 1 shared expert).
• ▸Context window: 1,048,576 tokens; maximum output 384K tokens.
• ▸Attention: hybrid Compressed Sparse Attention (CSA) plus Heavily Compressed Attention (HCA).
• ▸License: MIT, released April 24, 2026.
• ▸Native precision: mixed FP4 (MoE experts) plus FP8 (everything else), about 4.39 bits per weight.
• ▸Benchmarks: 86.2% MMLU-Pro, 88.1% GPQA Diamond, 56.9% Terminal Bench 2.0, 79.0% SWE-bench Verified.

This is where enthusiasm meets reality. DeepSeek-V4-Flash is a MoE giant, and even aggressive quantization leaves you well north of any single consumer card. Unsloth's dynamic GGUFs give you three practical tiers, and the native FP4/FP8 build sits among them. The numbers below are the published file sizes, not estimates.

• ▸UD-IQ3_XXS (3-bit) - 103 GB file, needs 110-135 GB of combined RAM plus VRAM. Unsloth calls this the best-value quant; it fits on a single 128 GB RAM device with CPU offload.
• ▸Native FP4 + FP8 - 146 GB. This is DeepSeek's own mixed-precision format (MoE experts in FP4, the rest in FP8), and the fastest option when your hardware supports FP4.
• ▸UD-Q4_K_XL (4-bit) - 162 GB file, needs roughly 162 GB of memory to hold comfortably.
• ▸UD-Q8_K_XL (near-lossless) - 162 GB file, about 169 GB minimum. Only worth it if you have server-class memory and want maximum fidelity.

Numbers beat promises. The clearest public measurement comes from a loftllc test running the native FP4/FP8 GGUF (146 GB, 4.39 bits per weight) on two RTX PRO 6000 Blackwell Max-Q cards - 96 GB each, 192 GB total - driven by an AMD EPYC 9175F with 768 GB of DDR5-6400. VRAM use landed at 75.1 GB on one card and 72.8 GB on the other.

That is roughly 36 tokens per second of generation for a 284B model - reading speed for a chat or agent loop, if not instant. And it is a floor, not a ceiling: the test ran with Flash Attention disabled, GPU utilization sitting at only 30-40%, and three graph splits, so the branch has real headroom as it matures. If you want the mechanics behind why active-parameter count, not total size, drives generation speed, our explainer on how local LLM inference actually works at /blog/how-local-llm-inference-actually-works walks through it.

No single consumer card runs this model - you are in multi-GPU or high-memory-workstation territory. Here is the realistic fit map, using combined VRAM (or unified memory) as the axis:

• ▸2x RTX PRO 6000 Blackwell (96 GB each): 192 GB total - runs the native FP4/FP8 build at ~36 tok/s. This is the reference setup.
• ▸2x H200 (141 GB each) or 4x A100 80GB: comfortably holds the FP4/FP8 build, which needs about 170 GB with overhead.
• ▸1x H200 141GB: fits the 3-bit Q3 GGUF (~103 GB) with room left for context.
• ▸Apple M3 Ultra 512GB unified: can load the 3-bit quant in unified memory, trading throughput for capacity.
• ▸Single RTX 5090 32GB or RTX 4090 24GB: does not fit - even the 103 GB quant needs heavy CPU offload and a lot of system RAM to run at single-digit tok/s.

If your target is a single 24 GB card, DeepSeek-V4-Flash is the wrong model - look at the smaller MoEs instead. Our guide on how much VRAM you need to run local LLMs at /blog/how-much-vram-to-run-local-llms sets realistic expectations tier by tier.

As of this writing, stock upstream llama.cpp cannot load DeepSeek-V4-Flash - the architecture needs a work-in-progress branch (wip/deepseek-v4-support, PR #22378) that adds native FP8 and MXFP4 handling. Build that branch first, then the simplest path is Unsloth's auto-download:

# Auto-download the recommended 3-bit dynamic quant and start chatting
export LLAMA_CACHE="unsloth/DeepSeek-V4-Flash-GGUF"

./llama.cpp/llama-cli \
    -hf unsloth/DeepSeek-V4-Flash-GGUF:UD-IQ3_XXS \
    --temp 1.0 \
    --top-p 1.0 \
    --min-p 0.0

If you have already downloaded the split GGUF shards, point llama-cli at the first shard and tune threads, context, and how many layers you push to the GPU. On a memory-constrained box you offload most of the model to system RAM:

# Manual load with CPU offload (most layers on RAM, a few on GPU)
./llama.cpp/llama-cli \
    --model ./DeepSeek-V4-Flash-UD-IQ3_XXS-00001-of-00004.gguf \
    --temp 1.0 \
    --top-p 1.0 \
    --min-p 0.0 \
    --threads 32 \
    --ctx-size 32768 \
    --n-gpu-layers 2

# Reference multi-GPU flags from the 192GB Blackwell run:
# --n-gpu-layers 999 --threads 15 --threads-batch 24 \
#   --ctx-size 8192 --parallel 1 -b 4096 -ub 2048

Flash is not the only oversized open-weight model competing for your VRAM this quarter. The real question is which giant earns the memory.

• ▸DeepSeek-V4-Flash - 284B total / 13B active, ~146 GB at FP4/FP8, 86.2% MMLU-Pro. The efficiency pick: the fewest active parameters for its quality tier.
• ▸GLM-5.2 - 744B parameters with a 1M context and Unsloth dynamic GGUFs available. A higher ceiling, but far heavier to host.
• ▸Kimi K3 - currently among the largest open-weight models shipped; more capability at the cost of extreme memory. We cover the fit math in Kimi K3 Is the Largest Open-Weight Model Ever at /blog/kimi-k3-largest-open-weight-model-run-locally.

For a side-by-side on which of these fits which hardware, our comparison GLM-5.2 vs DeepSeek V4 vs Kimi K3 at /blog/glm-5-2-vs-deepseek-v4-vs-kimi-k3-hardware-fit lays out the memory tiers directly.

With a model this large, the temptation is to push to the smallest quant that loads. Unsloth's dynamic quants hold up better than flat GGUF quantization because they protect the most sensitive layers, which is why the 3-bit UD-IQ3_XXS is usable at all on a MoE this size. Still, 3-bit is 3-bit - expect some drift on hard reasoning and long-context recall versus the FP4/FP8 build.

• ▸UD-IQ3_XXS (3-bit): the practical floor for single-workstation use, and the best fit-to-quality tradeoff Unsloth ships for this model.
• ▸Native FP4/FP8: the quality-and-speed sweet spot when your GPUs support FP4 math.
• ▸Q8 / near-lossless: reference quality, but 162 GB and up, and rarely worth it over FP4/FP8 for this model.

For a general feel of what each step down actually costs in quality, our breakdown Q4_K_M vs Q5_K_M vs Q6_K vs Q8_0 at /blog/q4-k-m-vs-q5-k-m-vs-q6-k-vs-q8-0-quantization-quality-loss measures the loss on smaller models where you can A/B test cheaply.

DeepSeek-V4-Flash sits at the extreme end of the local-model spectrum. Most builds are a simpler question of matching a model to the card you already own - and that is what Runyard is for.

• ▸VRAM Calculator (/tools/vram-calculator) - enter your GPU and system RAM to see which quants of a model fit and how fast they should run.
• ▸Compare (/compare) - put two GPUs or two models head-to-head on memory and throughput before you buy or download.
• ▸Model Radar (the Runyard home page) - track what Unsloth, Ollama, and the community just shipped, filtered to what your hardware can actually run.

Related reading on Runyard: how much VRAM you need to run local LLMs at /blog/how-much-vram-to-run-local-llms, the best GPU for local LLMs in 2026 at /blog/best-gpu-for-local-llms-2026, and how local LLM inference actually works at /blog/how-local-llm-inference-actually-works.