How slow is it if part of the model spills to system RAM?
Offloading a few layers to system RAM sounds like a small compromise. It is not — the slow side of the split dominates, and a fifth of the layers can cost half the speed.
The short answer
Computed from this tool’s default settings — your hardware and the rest as most people start. Change them below for your own case.
Everything fits on RTX 4090 — no offload, about 179 tok/s.
RAM Offload Speed Calculator
The real cost of running a model that does not quite fit.
Everything fits on RTX 4090 — no offload, about 179 tok/s.
| On GPU | Speed | Retained |
|---|---|---|
| 100% | 179.2 tok/s | 100% of full |
| 90% | 61.5 tok/s | 34% of full |
| 75% | 30.9 tok/s | 17% of full |
| 50% | 16.9 tok/s | 9% of full |
| 25% | 11.7 tok/s | 7% of full |
| 0% | 8.9 tok/s | 5% of full |
What each setting changes
Every input moves the result for a reason. This is what each one does and where to find the value for your own machine.
| Setting | Default | What it changes |
|---|---|---|
| Your hardware | RTX 4090 · 24 GB | The machine the model runs on. Usable memory decides what fits and memory bandwidth decides how fast it runs, so this moves every figure below. |
| Model size | 8B | 6 options, from 3B to 405B. |
| Quantisation | Q4_K_M — 4.5 bits/weight | 5 options, from Q8_0 — 8.5 bits/weight to Q3_K_M — 3.5 bits/weight. |
| Context length | 8192 tokens | Anywhere from 1,024 to 65,536 tokens. |
| System RAM bandwidth | 50 GB/s | Dual-channel DDR5 is roughly 60–90 GB/s; DDR4 nearer 40. |
Real answers across model size
The same calculation run at a range of settings, with everything else left at its default. These are computed by the tool itself, not written by hand.
| Model size | Effective speed | Layers on GPU | Fully on GPU | Speed retained |
|---|---|---|---|---|
| 3B | 477.9 tok/s | 100% | 478 tok/s | 100% |
| 8B | 179.2 tok/s | 100% | 179 tok/s | 100% |
| 14B | 102.4 tok/s | 100% | 102 tok/s | 100% |
| 32B | 44.8 tok/s | 100% | 45 tok/s | 100% |
| 70B | 1.7 tok/s | 43% | 20 tok/s | 8% |
Everything fits on RTX 4090 — no offload, about 478 tok/s.
How this is calculated
No lookup tables and no invented constants. Here is the arithmetic, so you can check it against your own numbers.
When a model does not fit, llama.cpp keeps some layers on the GPU and runs the rest on the CPU. Every token passes through all layers, so effective speed is a weighted harmonic mean of the two bandwidths — and the slow side dominates it.
That is why offload feels worse than the fraction suggests. Moving 20% of the layers to RAM does not cost 20%; on a fast card it can cost more than half the throughput.
These are well-founded engineering estimates, not benchmark results. Your quantisation, runtime and context length all move the real number, and usable memory is an assumption rather than a specification. See the full methodology for every assumption behind these figures.
Who this is for
The three situations that bring people to this calculation.
Model nearly fits
See the real speed before committing to offload.
Deciding
Compare offloading against a lower quantisation.
Diagnosing slowness
Confirm offload is why it crawls.
How to use this calculator
Four steps, no account, nothing leaves your browser.
Set your hardware
Start at the top of the panel. Every figure recalculates as you change it — there is no submit button, because watching the number move is the point.
Adjust the rest to match your setup
4 further settings: model size, quantisation, context length, system ram bandwidth. Defaults are the common case, so change only what differs for you.
Read the headline, then the table
The large figure answers the question. The table underneath shows how the answer changes across nearby settings, which is usually where the decision actually gets made.
Check it against the method
The arithmetic is written out above. If a number looks wrong for your hardware, the assumptions are the first place to look — usable memory and quantisation are the two that vary most.
How slow is it if part of the model spills to system RAM: common questions
The questions people ask about this, answered without hedging.
How much slower is CPU offload?
Far more than the fraction suggests. Every token passes through every layer, so the effective rate is a harmonic mean of the two bandwidths and the slow one dominates.
Is it better to offload or use a lower quant?
Almost always the lower quant. A model that fits entirely on the card at Q4 will beat the same model at Q5 with a fifth of its layers in system RAM, and by a wide margin.
Does faster RAM help?
Yes, roughly proportionally to the offloaded share — DDR5 over DDR4 is a real gain. It does not close the gap to VRAM, which is an order of magnitude faster.
Why is prompt processing still fast when generation is slow?
Prompt processing is compute bound and batches across tokens; generation is memory bound and reads all weights per token. Offload hurts the second far more.
Related local AI tools
All 50 run on the same arithmetic, so answers across them agree.
Will it fit
Context
Quantisation
Runtime & errors
Cost
Fine-tuning
Searches this page answers
Different ways of asking the same question, all resolved above.
Now find the models that fit
Sizing is only half the problem. Model Radar takes your hardware and shows which models actually run on it, ranked by what they are good at — the same arithmetic as this page, applied to every model worth running.