A 1M-Token Context Costs 147 GB of VRAM. Fable 5.1 Rents It.

Claude Fable 5.1 ships a 1 million-token context window, and the number sounds like a capability you could match locally with a big enough card. It is not. Holding a million tokens of context for Qwen3-8B — a model whose weights occupy about 4.5 GB at Q4_K_M — requires roughly 147 GB of VRAM for the KV cache alone. The context costs 33 times more than the model. Here is the arithmetic, and what your card can actually hold.

A model's weights are fixed. Load them once and they sit there, the same size whether you send ten tokens or ten thousand. The KV cache is different: it stores the key and value tensors for every token in the context, so it grows linearly with how much you have put in the window.

The size follows directly from the architecture. For every token you keep, the model stores two tensors per layer, sized by the number of key-value heads and the head dimension. Written out, that is two, times the layer count, times the KV heads, times the head dimension, times the number of tokens, times the bytes per value. Nothing in that expression is negotiable except the last two terms.

Qwen3-8B has 36 layers, 8 key-value heads and a head dimension of 128. At fp16 that works out to about 144 KB of cache for every single token. Multiply by a million and you have 147 GB — on a model you can otherwise run comfortably on an 8 GB card.

Those figures come from Qwen3-8B's own config.json, not an estimate. The same calculation on Qwen3-32B gives 262 GB at a million tokens, because it carries 64 layers instead of 36. Mistral-7B, with 32 layers, needs 131 GB.

Work backwards from VRAM. Take the card's capacity, subtract the model weights at your chosen quantisation, subtract about a gigabyte for runtime overhead, and divide what is left by the per-token cache cost. For Qwen3-8B at Q4_K_M that is roughly 4.5 GB of weights and 144 KB per token.

• ▸RTX 4060 Ti 16GB: about 10.5 GB free after weights and overhead, which holds roughly 71,000 tokens at fp16 cache.
• ▸RTX 4090 or 3090 (24 GB): about 18.5 GB free, or roughly 125,000 tokens.
• ▸RTX 5090 (32 GB): about 26.5 GB free, or roughly 180,000 tokens.
• ▸A full million tokens needs around 152 GB all in, which means datacentre hardware or several cards working together.

That is why a 24 GB card comfortably runs a model advertising a 128K window but struggles the moment you actually fill it. The model supports the window; your hardware decides how much of it you get to use.

It helps more than almost any other lever, because the cache scales linearly with bytes per value. Dropping the cache from fp16 to 8-bit halves it, and 4-bit halves it again. For Qwen3-8B at a million tokens that takes 147 GB down to about 74 GB at 8-bit and 37 GB at 4-bit.

Even 37 GB is beyond any consumer card, but at ordinary context lengths the same trick is transformative. A 128K window drops from 19.3 GB to about 4.8 GB at 4-bit, which turns an impossible configuration on a 16 GB card into a comfortable one. In llama.cpp the flags are --cache-type-k and --cache-type-v, and q8_0 is the setting most people should try first because the quality cost is small and the memory saving is immediate.

# 8-bit KV cache halves the context memory at negligible quality cost
llama-server -m qwen3-8b-Q4_K_M.gguf \
  --n-gpu-layers 999 \
  --ctx-size 131072 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

# Check what actually got allocated before assuming it fits
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

Filling a context also takes time. Prompt processing is compute-bound rather than bandwidth-bound, and it scales with how much you send. A consumer card prefilling at a few thousand tokens per second will spend several minutes simply reading a million-token prompt before it emits its first output token. That is time you pay on every request that does not hit a cache.

Attention cost compounds it. The memory figures above grow linearly with context, but the attention computation itself grows faster, which is why time-to-first-token degrades more sharply than the cache size suggests. In practice, long-context requests feel slow in a way the VRAM arithmetic alone does not predict, and that is true whether you are running locally or paying someone else to.

On the hosted side it is straightforward arithmetic. Fable 5.1 is priced at $10 per million input tokens, unchanged from Fable 5; only cache reads got cheaper, at $0.25 per million. Filling a million-token window once therefore costs roughly the price of a coffee, every time you send it fresh. Cache reads are the reason that number is survivable for agentic work: you pay the full rate once and a reduced rate on the reprocessing.

Locally the cost is not per-request, it is the hardware you would need to own. Holding that window on Qwen3-8B needs roughly 152 GB all in, which is several cards or a datacentre part. The comparison people reach for — cheap tokens versus a one-off GPU purchase — breaks down here, because at this context length there is no consumer GPU purchase that gets you to the same place at any price.

There is a second limit that gets lost in the marketing. Qwen3-8B and Qwen3-32B both declare a maximum position embedding of 40,960 tokens in their configuration. That is the window the model was actually trained to handle. Pushing beyond it requires RoPE scaling, and quality degrades as you stretch further from what the model saw in training.

So the honest comparison is not "1M hosted versus 1M local". It is a hosted million-token window against a local window that is architecturally capped well below that and practically capped lower still by your VRAM. If your workload genuinely needs to reason over a million tokens in one pass, that is one of the clearest remaining cases for a hosted model.

Measure before you set it. Most people choose a context length by taking the largest number the model advertises, then discover it will not load. The better order is to find the longest input you genuinely send, add headroom for the reply, and set the window just above that. A codebase question that ships 12,000 tokens does not need a 128K window, and the difference is gigabytes.

Those gigabytes have somewhere better to go. On a 16 GB card, dropping from a 128K window to 32K on Qwen3-8B frees roughly 14 GB, which is the difference between running the model at Q4 and running it at Q8 with room to spare. Higher quantisation improves every response you generate; an unused context window improves nothing.

Mixture-of-experts models are worth a separate note. Their cache scales with active parameters rather than total, so an MoE model that fills your card with weights can still leave usable room for context. It is one of the few places the architecture helps twice: fewer active parameters mean both faster generation and a smaller cache per token.

Most workloads that appear to need an enormous context actually need retrieval. Feeding an entire codebase into the window is expensive locally and expensive hosted; retrieving the twenty relevant files and sending those is cheaper on both, and usually produces better answers because the model is not diluting attention across irrelevant material.

Where you do need long context, set it deliberately rather than maximally. Asking for the full declared window when your documents are 30K tokens wastes gigabytes that could hold a larger quantisation of the model itself, and a bigger model at Q5 will generally serve you better than a smaller one at Q4 with a window you never fill.

Our <a href="/models">Model Explorer</a> computes weights and cache together for any model, quantisation and context length, so you can see the trade directly. The <a href="/tools/context-window-memory-calculator">context window memory calculator</a> isolates just the cache term if that is the part you are tuning, and the <a href="/gpu">GPU pages</a> list what each card holds. For the pricing side of the Fable 5.1 release, see our <a href="/blog/fable-5-1-vs-fable-5-what-changed">breakdown of what actually changed</a>.

One last practical note: the window you configure is allocated, not borrowed. llama.cpp and vLLM both reserve the cache up front, so asking for 128K and using 8K still costs you the full 128K in VRAM for the whole session. That is why a model can load fine and then fail on a longer conversation only if the window was set too small, and why it can fail to load at all when the window was set too large. Neither failure is about the weights.