CUDA Out of Memory: The Five Causes, in Order

CUDA out of memory is the most common error in local AI and it has a short list of causes. Working through them in the right order saves a lot of time.

Start with the number that explains most cases. A 14B model at Q4_K_M with an 8K context needs about 10.71 GB. A 12 GB card gives you about 10.8 GB usable. It fits — by 0.09 GB.

If it fails immediately on load, the model is simply too big. No amount of tuning fixes this; you need a smaller model or a lower quantisation.

If it loaded fine and died an hour into a conversation, this is it. The cache grows with every token and does not shrink until you clear the chat. Starting a new conversation releases it immediately.

Browsers, video calls, the desktop compositor, and an earlier model you forgot to unload. Check with nvidia-smi before blaming the model.

Long-running processes that repeatedly allocate and free can leave memory that is technically available but not contiguous. Restarting the runtime fixes it, which is why 'have you tried restarting' is annoyingly effective.

Serving frameworks pre-allocate a pool. vLLM's gpu-memory-utilization defaults high on purpose, and if anything else wants memory afterwards there is none left.

On that 14B at Q4_K_M on a 12 GB card:

• ▸Drop to Q3_K_M — frees about 1.75 GB. The biggest single lever.
• ▸Halve the context from 8K to 4K — frees about 0.92 GB.
• ▸Both together — takes you from 10.71 GB to 8.04 GB, a comfortable margin.
• ▸Flash attention — smaller, but free, and grows with context.
• ▸Quantise the KV cache to 8-bit — roughly halves the cache term.

Notice the ordering. Quantisation frees nearly twice what halving your context does, and costs less in practice than losing half your window.

Every figure here is computed the same way the calculators do it: weights are parameters times bits-per-weight divided by eight, plus a KV-cache term for your context, plus about a gigabyte of runtime overhead. Usable memory is roughly 90% of nameplate on a discrete card.