How much memory does flash attention save?
Flash attention is the rare optimisation with no downside: less peak memory, faster at long context, and identical output. The only question is how much it buys you.
The short answer
Computed from this tool’s default settings — model size and the rest as most people start. Change them below for your own case.
Flash attention takes your ceiling from 97,280 to 121,856 tokens on RTX 4090 — 0.02 GB back at your current setting. It costs nothing in quality; turn it on.
Flash Attention Savings Calculator
What --flash-attn buys you in context length, on your card.
Flash attention takes your ceiling from 97,280 to 121,856 tokens on RTX 4090 — 0.02 GB back at your current setting. It costs nothing in quality; turn it on.
llama-server -m model.gguf --flash-attn
What each setting changes
Every input moves the result for a reason. This is what each one does and where to find the value for your own machine.
| Setting | Default | What it changes |
|---|---|---|
| Model size | 8B | 6 options, from 3B to 405B. |
| Context length | 8192 tokens | Anywhere from 1,024 to 131,072 tokens. |
| Your hardware | RTX 4090 · 24 GB | The machine the model runs on. Usable memory decides what fits and memory bandwidth decides how fast it runs, so this moves every figure below. |
Real answers across model size
The same calculation run at a range of settings, with everything else left at its default. These are computed by the tool itself, not written by hand.
| Model size | Memory saved | Without | With flash attention | Max context without |
|---|---|---|---|---|
| 3B | 0.01 GB | 3.54 GB | 3.53 GB | 186,368 |
| 8B | 0.02 GB | 6.89 GB | 6.87 GB | 97,280 |
| 14B | 0.02 GB | 10.7 GB | 10.7 GB | 58,368 |
| 32B | 0.03 GB | 21.8 GB | 21.7 GB | 8,192 |
| 70B | 0.05 GB | 44.5 GB | 44.4 GB | 0 |
Flash attention takes your ceiling from 186,368 to 232,448 tokens on RTX 4090 — 0.01 GB back at your current setting. It costs nothing in quality; turn it on.
How this is calculated
No lookup tables and no invented constants. Here is the arithmetic, so you can check it against your own numbers.
Flash attention never materialises the full attention matrix, computing it in tiles instead. The saving is in the attention workspace rather than in the cache itself, and it grows with context — negligible at 2k, substantial past 32k, which is where people notice it.
The estimate treats the saving as roughly a fifth of cache-scale memory at long context, tapering to nothing at short context. It costs no quality, so if your card supports it there is no reason not to enable it.
These are well-founded engineering estimates, not benchmark results. Your quantisation, runtime and context length all move the real number, and usable memory is an assumption rather than a specification. See the full methodology for every assumption behind these figures.
Who this is for
The three situations that bring people to this calculation.
Long context
See how much further you can push the window.
Near the limit
Find whether it is enough to avoid an OOM.
Deciding to enable
Confirm there is nothing to trade away.
How to use this calculator
Four steps, no account, nothing leaves your browser.
Set model size
Start at the top of the panel. Every figure recalculates as you change it — there is no submit button, because watching the number move is the point.
Adjust the rest to match your setup
2 further settings: context length, your hardware. Defaults are the common case, so change only what differs for you.
Read the headline, then the table
The large figure answers the question. The table underneath shows how the answer changes across nearby settings, which is usually where the decision actually gets made.
Check it against the method
The arithmetic is written out above. If a number looks wrong for your hardware, the assumptions are the first place to look — usable memory and quantisation are the two that vary most.
How much memory does flash attention save: common questions
The questions people ask about this, answered without hedging.
What is flash attention?
An implementation that computes attention in tiles inside fast on-chip memory instead of materialising the full attention matrix. Same mathematics, same output, less memory traffic.
Does flash attention change output quality?
No. It is an exact reformulation, not an approximation, so results are identical up to floating-point ordering.
Should I always enable it?
Yes where it is supported. The saving grows with context and there is no trade-off to weigh — in llama.cpp it is a single --flash-attn flag.
Why does it barely help at short context?
The attention workspace it avoids scales with sequence length. At 2K there is little to save; past 32K it becomes substantial.
Related local AI tools
All 50 run on the same arithmetic, so answers across them agree.
More on context
Will it fit
Speed
Quantisation
Runtime & errors
Cost
Fine-tuning
Searches this page answers
Different ways of asking the same question, all resolved above.
Now find the models that fit
Sizing is only half the problem. Model Radar takes your hardware and shows which models actually run on it, ranked by what they are good at — the same arithmetic as this page, applied to every model worth running.