Which quantisation should I use?
Quantisation is the one lever that reliably brings a model into range, and the ladder has a clear sweet spot: Q4_K_M holds almost all the quality for less than half the size.
The short answer
Computed from this tool’s default settings — your hardware and the rest as most people start. Change them below for your own case.
Q8_0 is the highest fidelity that fits RTX 4090, using 10.9 GB of 22.1 GB.
Quantisation Picker
The highest-fidelity quantisation your hardware can actually hold.
Q8_0 is the highest fidelity that fits RTX 4090, using 10.9 GB of 22.1 GB.
| Quantisation | Total needed | Verdict | Trade-off |
|---|---|---|---|
| Q8_0 | 10.9 GB | Fits | Near-lossless. Use when memory is free. |
| Q6_K | 8.99 GB | Fits | Very close to FP16; the safe default if it fits. |
| Q5_K_M | 8.09 GB | Fits | Mild loss, noticeably smaller. |
| Q4_K_M | 6.89 GB | Fits | The sweet spot most people run. |
| Q3_K_M | 5.89 GB | Fits | Visible degradation. Fallback only. |
What each setting changes
Every input moves the result for a reason. This is what each one does and where to find the value for your own machine.
| Setting | Default | What it changes |
|---|---|---|
| Your hardware | RTX 4090 · 24 GB | The machine the model runs on. Usable memory decides what fits and memory bandwidth decides how fast it runs, so this moves every figure below. |
| Model size | 8B — Llama 3.1 8B class | Capacity follows total parameters, even for mixture-of-experts models. |
| Context length | 8192 tokens | The KV cache grows linearly with this. It is the biggest lever you have. |
Real answers across model size
The same calculation run at a range of settings, with everything else left at its default. These are computed by the tool itself, not written by hand.
| Model size | Pick this | Usable memory | Uses |
|---|---|---|---|
| 1B — Llama 3.2 1B class | Q8_0 | 22.1 GB | 2.55 GB |
| 7B — Mistral / Qwen 7B class | Q8_0 | 22.1 GB | 9.74 GB |
| 14B — Qwen 14B class | Q8_0 | 22.1 GB | 17.7 GB |
| 32B — Qwen 32B class | Q4_K_M | 22.1 GB | 21.8 GB |
| 120B — gpt-oss 120B class | Nothing fits | 22.1 GB |
Q8_0 is the highest fidelity that fits RTX 4090, using 2.55 GB of 22.1 GB.
How this is calculated
No lookup tables and no invented constants. Here is the arithmetic, so you can check it against your own numbers.
Quantisation trades precision for size. Q4_K_M is the point where the loss stops being noticeable for most work, which is why it is the common default. Q3_K_M and below show visible degradation and should be treated as a fallback rather than a choice.
The rule that holds across model families: a larger model at Q4 generally beats a smaller model at Q8. Spend spare memory on parameters before precision.
These are well-founded engineering estimates, not benchmark results. Your quantisation, runtime and context length all move the real number, and usable memory is an assumption rather than a specification. See the full methodology for every assumption behind these figures.
Who this is for
The three situations that bring people to this calculation.
Model does not fit
Find the highest quality level that does.
Trading speed
See what a lower rung buys in tokens per second.
Quality first
Check whether Q6 or Q8 is affordable on your card.
How to use this calculator
Four steps, no account, nothing leaves your browser.
Set your hardware
Start at the top of the panel. Every figure recalculates as you change it — there is no submit button, because watching the number move is the point.
Adjust the rest to match your setup
2 further settings: model size, context length. Defaults are the common case, so change only what differs for you.
Read the headline, then the table
The large figure answers the question. The table underneath shows how the answer changes across nearby settings, which is usually where the decision actually gets made.
Check it against the method
The arithmetic is written out above. If a number looks wrong for your hardware, the assumptions are the first place to look — usable memory and quantisation are the two that vary most.
Which quantisation should I use: common questions
The questions people ask about this, answered without hedging.
Which quantisation should I use?
Q4_K_M unless you have memory to spare. It is the widely accepted sweet spot: roughly half the size of the 8-bit build with quality differences most people cannot detect in normal use.
How much quality do I lose at Q4?
Little, for a model of reasonable size. The loss grows sharply below 4 bits — Q3 is noticeably weaker and Q2 often breaks reasoning outright. Prefer a smaller model at Q4 over a larger one at Q2.
What does the K in Q4_K_M mean?
K-quants are a smarter scheme that varies precision within each block and keeps sensitive tensors higher. The suffix is the size variant: S small, M medium, L large. M is the usual choice.
Is Q8 worth it over Q6?
Rarely. Q6_K is already close to indistinguishable from full precision for most work, and Q8 costs about 30% more memory for a difference that is hard to measure.
Related local AI tools
All 50 run on the same arithmetic, so answers across them agree.
More on quantisation
Will it fit
Context
Speed
Runtime & errors
Cost
Fine-tuning
Searches this page answers
Different ways of asking the same question, all resolved above.
Now find the models that fit
Sizing is only half the problem. Model Radar takes your hardware and shows which models actually run on it, ranked by what they are good at — the same arithmetic as this page, applied to every model worth running.