How many layers can I offload to my GPU?
When a model does not fit, llama.cpp keeps what it can on the GPU and runs the rest on the CPU. Getting --n-gpu-layers right is the difference between usable and unusable.
The short answer
Computed from this tool’s default settings — your hardware and the rest as most people start. Change them below for your own case.
Everything fits — use --n-gpu-layers 999 and keep the whole model on the card.
CPU vs GPU Offload Calculator
The --n-gpu-layers value that fits, instead of guessing until it loads.
Everything fits — use --n-gpu-layers 999 and keep the whole model on the card.
llama-server -m model-Q4_K_M.gguf \ --n-gpu-layers 999 \ --ctx-size 8192 # 999 means "offload everything"
What each setting changes
Every input moves the result for a reason. This is what each one does and where to find the value for your own machine.
| Setting | Default | What it changes |
|---|---|---|
| Your hardware | RTX 4090 · 24 GB | The machine the model runs on. Usable memory decides what fits and memory bandwidth decides how fast it runs, so this moves every figure below. |
| Model size | 8B — Llama 3.1 8B class | Capacity follows total parameters, even for mixture-of-experts models. |
| Quantisation | Q4_K_M — 4.5 bits/weight | 5 options, from Q8_0 — 8.5 bits/weight to Q3_K_M — 3.5 bits/weight. |
| Context length | 8192 tokens | The KV cache grows linearly with this. It is the biggest lever you have. |
Real answers across model size
The same calculation run at a range of settings, with everything else left at its default. These are computed by the tool itself, not written by hand.
| Model size | Layers that fit on the GPU | Estimated layers | Per layer | Budget after cache |
|---|---|---|---|---|
| 1B — Llama 3.2 1B class | All 17 | 17 | 0.03 GB | 20.6 GB |
| 7B — Mistral / Qwen 7B class | All 32 | 32 | 0.12 GB | 19.8 GB |
| 14B — Qwen 14B class | All 40 | 40 | 0.20 GB | 19.2 GB |
| 32B — Qwen 32B class | All 53 | 53 | 0.34 GB | 18.3 GB |
| 120B — gpt-oss 120B class | 19 of 83 | 83 | 0.81 GB | 15.7 GB |
Everything fits — use --n-gpu-layers 999 and keep the whole model on the card.
How this is calculated
No lookup tables and no invented constants. Here is the arithmetic, so you can check it against your own numbers.
llama.cpp puts as many transformer layers on the GPU as you tell it to with --n-gpu-layers, and keeps the rest on the CPU. Too many and it fails at load time with a CUDA out-of-memory error; too few and you leave speed on the table.
Layer counts are not published per model, so the figure below is estimated from the parameter count using the shape transformer families actually use. Treat it as a starting point: if it fails to load, drop two or three and retry.
These are well-founded engineering estimates, not benchmark results. Your quantisation, runtime and context length all move the real number, and usable memory is an assumption rather than a specification. See the full methodology for every assumption behind these figures.
Who this is for
The three situations that bring people to this calculation.
Model slightly too big
Find the layer count that loads without crashing.
Tuning by hand
Skip the trial-and-error loop of stepping the number down.
Deciding
See whether offloading beats dropping a quantisation rung.
How to use this calculator
Four steps, no account, nothing leaves your browser.
Set your hardware
Start at the top of the panel. Every figure recalculates as you change it — there is no submit button, because watching the number move is the point.
Adjust the rest to match your setup
3 further settings: model size, quantisation, context length. Defaults are the common case, so change only what differs for you.
Read the headline, then the table
The large figure answers the question. The table underneath shows how the answer changes across nearby settings, which is usually where the decision actually gets made.
Check it against the method
The arithmetic is written out above. If a number looks wrong for your hardware, the assumptions are the first place to look — usable memory and quantisation are the two that vary most.
How many layers can I offload to my GPU: common questions
The questions people ask about this, answered without hedging.
What is --n-gpu-layers?
How many transformer layers to place on the GPU; the rest run on the CPU. Setting it high enough to hold the whole model is always fastest, and 999 is the usual shorthand for "all of them".
How many layers should I offload?
As many as fit after the KV cache. If loading fails, drop by two or three and retry — layer counts vary between models, so the estimate is a starting point.
Is partial offload worth it?
Often not. Every token passes through every layer, so the slow side dominates: moving 20% of layers to RAM can cost more than half your throughput. A lower quantisation that fits entirely on the card is usually faster.
Does more system RAM help?
It lets a bigger model load, but does not make it fast. Speed follows the bandwidth of whichever memory holds the weights, and system RAM is roughly ten times slower than a modern card.
Related local AI tools
All 50 run on the same arithmetic, so answers across them agree.
More on runtime & errors
Will it fit
Context
Speed
Quantisation
Cost
Fine-tuning
Searches this page answers
Different ways of asking the same question, all resolved above.
Now find the models that fit
Sizing is only half the problem. Model Radar takes your hardware and shows which models actually run on it, ranked by what they are good at — the same arithmetic as this page, applied to every model worth running.