How much memory does the KV cache use?
Weights are allocated once and never move. The KV cache grows with every token in the conversation, which is why it — not the model — is usually what runs you out of memory.
The short answer
Computed from this tool’s default settings — model size and the rest as most people start. Change them below for your own case.
At 8,192 tokens the cache is 1.39 GB, which is 24% of what the model occupies. This is the term that grows while you work, so leave room for it rather than filling the card with weights.
KV Cache Calculator
The part of your VRAM that grows with every token in the conversation.
At 8,192 tokens the cache is 1.39 GB, which is 24% of what the model occupies. This is the term that grows while you work, so leave room for it rather than filling the card with weights.
| Context | Cache | Total | On your card |
|---|---|---|---|
| 2,048 | 0.35 GB | 5.85 GB | Fits |
| 8,192 | 1.39 GB | 6.89 GB | Fits |
| 32,768 | 5.56 GB | 11.1 GB | Fits |
| 65,536 | 11.1 GB | 16.6 GB | Fits |
| 131,072 | 22.2 GB | 27.7 GB | Too big |
What each setting changes
Every input moves the result for a reason. This is what each one does and where to find the value for your own machine.
| Setting | Default | What it changes |
|---|---|---|
| Model size | 8B | 6 options, from 3B to 405B. |
| Context length | 8192 tokens | Anywhere from 1,024 to 131,072 tokens. |
| Your hardware | RTX 4090 · 24 GB | The machine the model runs on. Usable memory decides what fits and memory bandwidth decides how fast it runs, so this moves every figure below. |
Real answers across model size
The same calculation run at a range of settings, with everything else left at its default. These are computed by the tool itself, not written by hand.
| Model size | KV cache at this context | Weights | Cache | Cache share |
|---|---|---|---|---|
| 3B | 0.85 GB | 1.69 GB | 0.85 GB | 34% |
| 8B | 1.39 GB | 4.50 GB | 1.39 GB | 24% |
| 14B | 1.84 GB | 7.88 GB | 1.84 GB | 19% |
| 32B | 2.78 GB | 18.0 GB | 2.78 GB | 13% |
| 70B | 4.11 GB | 39.4 GB | 4.11 GB | 9% |
At 8,192 tokens the cache is 0.85 GB, which is 34% of what the model occupies. This is the term that grows while you work, so leave room for it rather than filling the card with weights.
How this is calculated
No lookup tables and no invented constants. Here is the arithmetic, so you can check it against your own numbers.
The KV cache stores one key and one value vector per token per layer, so it grows linearly with context length and does not shrink until the conversation is cleared. Weights are fixed; the cache is what actually moves as you use the model.
The estimate scales with the square root of parameter count, which tracks how transformer families widen as they get deeper, and is the same formula used across the rest of the site so the numbers agree.
These are well-founded engineering estimates, not benchmark results. Your quantisation, runtime and context length all move the real number, and usable memory is an assumption rather than a specification. See the full methodology for every assumption behind these figures.
Who this is for
The three situations that bring people to this calculation.
Crashes mid-session
See the cache growth that caused it.
Long context
Budget the cache before setting a context length.
Serving
Cost a session so you can set a concurrency limit.
How to use this calculator
Four steps, no account, nothing leaves your browser.
Set model size
Start at the top of the panel. Every figure recalculates as you change it — there is no submit button, because watching the number move is the point.
Adjust the rest to match your setup
2 further settings: context length, your hardware. Defaults are the common case, so change only what differs for you.
Read the headline, then the table
The large figure answers the question. The table underneath shows how the answer changes across nearby settings, which is usually where the decision actually gets made.
Check it against the method
The arithmetic is written out above. If a number looks wrong for your hardware, the assumptions are the first place to look — usable memory and quantisation are the two that vary most.
How much memory does the KV cache use: common questions
The questions people ask about this, answered without hedging.
Why did my model fit at first and then run out of memory?
Almost always the KV cache. Weights are allocated once at load; the cache grows with every token you add, so a long conversation can push a model that loaded cleanly into an out-of-memory error hours later.
What is the KV cache?
Stored key and value vectors for every token processed so far, one pair per layer. It lets the model attend to earlier tokens without recomputing them, at the cost of memory that grows linearly with the conversation.
How do I reduce the KV cache?
Shorten the context, enable flash attention, or quantise the cache — llama.cpp supports 8-bit and 4-bit KV cache, which halves or quarters it for a modest quality cost on long contexts.
Does clearing the chat free the memory?
Yes. Starting a new conversation releases the cache; the weights stay loaded. It is the quickest fix when a long session starts pressing against the limit.
Why is the cache bigger on some models than others?
It scales with layer count and hidden size, and with how many key/value heads the architecture uses. Grouped-query attention shares them across heads and shrinks the cache substantially, which is why modern models handle long context better.
Related local AI tools
All 50 run on the same arithmetic, so answers across them agree.
More on context
Will it fit
Speed
Quantisation
Runtime & errors
Cost
Fine-tuning
Searches this page answers
Different ways of asking the same question, all resolved above.
Now find the models that fit
Sizing is only half the problem. Model Radar takes your hardware and shows which models actually run on it, ranked by what they are good at — the same arithmetic as this page, applied to every model worth running.