Why am I getting CUDA out of memory, and how do I fix it?
CUDA out of memory is the most common error in local AI, and it has a short list of causes. This works out which one you hit and what to change.
The short answer
Computed from this tool’s default settings — your hardware and the rest as most people start. Change them below for your own case.
These settings fit in 22.1 GB with 15.2 GB to spare. If you are still seeing OOM, something else is holding VRAM — check nvidia-smi for a stale process.
OOM Fix Assistant
Work out which lever to pull — context, layers or quantisation — and by how much.
These settings fit in 22.1 GB with 15.2 GB to spare. If you are still seeing OOM, something else is holding VRAM — check nvidia-smi for a stale process.
What each setting changes
Every input moves the result for a reason. This is what each one does and where to find the value for your own machine.
| Setting | Default | What it changes |
|---|---|---|
| Your hardware | RTX 4090 · 24 GB | The machine the model runs on. Usable memory decides what fits and memory bandwidth decides how fast it runs, so this moves every figure below. |
| Model size | 8B — Llama 3.1 8B class | Capacity follows total parameters, even for mixture-of-experts models. |
| Quantisation you tried | Q4_K_M — 4.5 bits/weight | 5 options, from Q8_0 — 8.5 bits/weight to Q3_K_M — 3.5 bits/weight. |
| Context length | 8192 tokens | The KV cache grows linearly with this. It is the biggest lever you have. |
Real answers across model size
The same calculation run at a range of settings, with everything else left at its default. These are computed by the tool itself, not written by hand.
| Model size | Diagnosis | You have | This needs | KV cache |
|---|---|---|---|---|
| 1B — Llama 3.2 1B class | This should load | 22.1 GB | 2.05 GB | 0.49 GB |
| 7B — Mistral / Qwen 7B class | This should load | 22.1 GB | 6.24 GB | 1.30 GB |
| 14B — Qwen 14B class | This should load | 22.1 GB | 10.7 GB | 1.84 GB |
| 32B — Qwen 32B class | This should load | 22.1 GB | 21.8 GB | 2.78 GB |
| 120B — gpt-oss 120B class | 51.8 GB | 22.1 GB | 73.9 GB | 5.38 GB |
These settings fit in 22.1 GB with 20.0 GB to spare. If you are still seeing OOM, something else is holding VRAM — check nvidia-smi for a stale process.
How this is calculated
No lookup tables and no invented constants. Here is the arithmetic, so you can check it against your own numbers.
A CUDA out-of-memory error at load time means weights plus KV cache plus attention buffers exceeded your VRAM. There are three levers, and they are not equal: context size is usually the biggest, then GPU layers, then quantisation.
Reduce context first. The KV cache grows linearly with it, so halving the context frees far more than stepping down a quantisation level, and costs less quality.
These are well-founded engineering estimates, not benchmark results. Your quantisation, runtime and context length all move the real number, and usable memory is an assumption rather than a specification. See the full methodology for every assumption behind these figures.
Who this is for
The three situations that bring people to this calculation.
Crash on load
The weights never fit — find what does.
Crash mid-conversation
The cache grew past the headroom you had left.
Intermittent crashes
You are running at the edge; find the safe setting.
How to use this calculator
Four steps, no account, nothing leaves your browser.
Set your hardware
Start at the top of the panel. Every figure recalculates as you change it — there is no submit button, because watching the number move is the point.
Adjust the rest to match your setup
3 further settings: model size, quantisation you tried, context length. Defaults are the common case, so change only what differs for you.
Read the headline, then the table
The large figure answers the question. The table underneath shows how the answer changes across nearby settings, which is usually where the decision actually gets made.
Check it against the method
The arithmetic is written out above. If a number looks wrong for your hardware, the assumptions are the first place to look — usable memory and quantisation are the two that vary most.
Why am I getting CUDA out of memory, and how do I fix it: common questions
The questions people ask about this, answered without hedging.
Why do I get CUDA out of memory?
Something asked for more VRAM than remained. On load it is the weights; mid-session it is almost always the KV cache growing with the conversation; sometimes it is another process, since a browser or a desktop compositor can hold a surprising amount.
How do I fix out of memory without new hardware?
In order of effect: drop one quantisation rung, shorten the context, enable flash attention, quantise the KV cache, then offload a few layers to the CPU. The first two solve most cases.
Why does it crash after working fine for an hour?
The cache grows as the conversation does. A model that loads with a gigabyte spare will consume that gigabyte over a long session. Set a context you can afford up front.
Does closing other applications help?
Sometimes materially. Browsers, video calls and the desktop compositor all hold VRAM. Check with nvidia-smi before assuming the model is at fault.
Related local AI tools
All 50 run on the same arithmetic, so answers across them agree.
More on runtime & errors
Will it fit
Context
Speed
Quantisation
Cost
Fine-tuning
Searches this page answers
Different ways of asking the same question, all resolved above.
Now find the models that fit
Sizing is only half the problem. Model Radar takes your hardware and shows which models actually run on it, ranked by what they are good at — the same arithmetic as this page, applied to every model worth running.