Qwen3.8-27B's KV Cache Is 4x Smaller Than the Formula Says

In our Qwen3.8-27B post we flagged something we could not explain. Users reported running 160,000-token contexts on a single 24 GB card, and our arithmetic said that was impossible by a wide margin. We told you to trust the real runs over our estimate. Here is why they were right.

Qwen3.8-27B does not use full attention throughout. It uses a hybrid arrangement: three blocks of gated DeltaNet linear attention for every one block of full attention, across 64 layers. Linear attention carries a constant recurrent state rather than a growing cache.

So only 16 of the 64 layers keep a KV cache at all. Reported figures put it at about 64 KB per token, against roughly 256 KB per token for a conventional 64-layer model of the same size — four times smaller.

Recomputing with only the full-attention layers counted, at Q4_K_M where the weights are about 15.2 GB:

• ▸32K context — about 2.6 GB of cache, 18.7 GB total. Fits a 24 GB card.
• ▸64K context — about 5.1 GB of cache, 21.3 GB total. Still fits.
• ▸128K context — about 10.2 GB, 26.4 GB total. Needs 32 GB.
• ▸262K context — about 20.4 GB, 36.6 GB total.

Our own formula, which scales with parameter count and assumes every layer caches, gave 20.4 GB of cache at 64K where the real figure is nearer 5. That is the difference between “impossible on a 24 GB card” and “comfortable”.

Full attention costs memory that grows linearly with context and compute that grows with its square. Linear attention replaces the growing cache with a fixed-size recurrent state, which makes long context affordable but is generally weaker at precise recall over long distances.

Hybrid designs take both: linear attention for most layers to keep the cost down, full attention on a minority of layers to keep the recall. It is a pragmatic trade and it is why a 262K window on a 27B model stopped being absurd.

• ▸Check the architecture before trusting any KV cache estimate, including ours. Hybrid and full attention differ by multiples, not percentages.
• ▸A long context window on a recent model is not automatically expensive. It was, on models designed three years ago.
• ▸Weights are still weights. Hybrid attention changes the cache term and nothing else — a 27B model is still about 15 GB at Q4_K_M.
• ▸If a model advertises a very large context and modest hardware, look for linear or hybrid attention. That is usually what makes it possible.

We are fixing the estimate in our tools. Until then, treat our long-context figures as an upper bound for hybrid models, and as roughly right for conventional ones.

You do not have to take our word for the layer split, and you should not — the difference between hybrid and full attention is a multiple, not a rounding error. The model's own configuration states it.

# Pull just the config from the model repo
curl -s https://huggingface.co/<org>/<model>/raw/main/config.json | \
  python -m json.tool | grep -iE 'layer|attention|head|window'

# What to look for:
#   num_hidden_layers        total layers
#   full_attention_interval  1 layer in N keeps a KV cache
#   num_key_value_heads      grouped-query attention if < num_attention_heads
#   sliding_window           attention capped at a window rather than global

For Qwen3.8-27B, full_attention_interval is 4 across 64 layers, which is where the 16 comes from. If a model has no such field, assume every layer caches and our estimate is roughly right.

Hybrid attention is the newest of several tricks, and they stack. Knowing which a model uses tells you how far to trust any generic estimate:

Grouped-query attention

Multiple attention heads share one set of key and value heads. If a model lists 8 key-value heads against 64 attention heads, the cache is eight times smaller than a naive calculation suggests. Almost every model released since 2024 does this, which is the main reason our formula runs conservative even for conventional architectures.

Sliding-window attention

Attention is capped at a fixed window rather than the whole context, so the cache stops growing once the window fills. A model with a 4K sliding window has a constant cache cost no matter how long the conversation gets.

Hybrid and linear attention

What Qwen3.8-27B does. Most layers carry a fixed-size recurrent state instead of a cache, and only a minority keep full attention. This is the largest single reduction of the three, and the newest.

The practical consequence is that the old rule — long context needs a big card — is weakening. On a conventional 27B, a 64K context genuinely cost you 20 GB of cache and put the model out of reach of a 24 GB card. On this one it costs about 5 GB and fits comfortably.

So if you were about to buy more memory specifically to get a longer context, check the architecture first. It may already be affordable on the card you have, and the money is better spent on capacity for a larger model.

Does hybrid attention make the model worse?

Linear attention is generally weaker at precise recall over long distances than full attention. That is exactly why these designs keep a minority of full-attention layers rather than going fully linear — the hybrid is a deliberate compromise between cost and recall, not a free win.

Do the weights change?

No. Hybrid attention affects the cache term only. A 27B model is still about 15.2 GB at Q4_K_M, and that number is unaffected by how the layers attend.

Will llama.cpp and Ollama handle it correctly?

They need explicit support for the architecture, which is why a brand-new model sometimes needs the very latest build. If a hybrid model loads but uses far more memory than expected, an older runtime falling back to full attention for every layer is a likely cause.

Is our calculator wrong for every model then?

It is conservative — it over-estimates rather than under-estimates, which is the safer direction to be wrong in. For conventional models it is close. For grouped-query models it runs high. For hybrid models at long context it can be several times high, as here.