How do model sizes compare on my hardware?
Model sizes are easier to judge side by side than one at a time. This lays out what each size costs in memory and returns in speed on your hardware.
The short answer
Computed from this tool’s default settings — your hardware and the rest as most people start. Change them below for your own case.
At Q4_K_M and 8,192 tokens of context, RTX 4090 holds 7 of the 10 size classes. Speed falls as size rises because each token streams more weights.
Model Comparison Matrix
Memory and speed for every size class on one device, side by side.
At Q4_K_M and 8,192 tokens of context, RTX 4090 holds 7 of the 10 size classes. Speed falls as size rises because each token streams more weights.
| Size | Weights | Total | Speed | Verdict |
|---|---|---|---|---|
| 1B | 0.56 GB | 2.05 GB | 1,434 tok/s | Fits |
| 3B | 1.69 GB | 3.54 GB | 478 tok/s | Fits |
| 7B | 3.94 GB | 6.24 GB | 205 tok/s | Fits |
| 8B | 4.50 GB | 6.89 GB | 179 tok/s | Fits |
| 14B | 7.88 GB | 10.7 GB | 102 tok/s | Fits |
| 27B | 15.2 GB | 18.7 GB | 53.1 tok/s | Fits |
| 32B | 18.0 GB | 21.8 GB | 44.8 tok/s | Fits |
| 70B | 39.4 GB | 44.5 GB | 20.5 tok/s | Too big |
| 120B | 67.5 GB | 73.9 GB | 11.9 tok/s | Too big |
| 405B | 227.8 GB | 238.7 GB | 3.5 tok/s | Too big |
What each setting changes
Every input moves the result for a reason. This is what each one does and where to find the value for your own machine.
| Setting | Default | What it changes |
|---|---|---|
| Your hardware | RTX 4090 · 24 GB | The machine the model runs on. Usable memory decides what fits and memory bandwidth decides how fast it runs, so this moves every figure below. |
| Quantisation | Q4_K_M — 4.5 bits/weight | 5 options, from Q8_0 — 8.5 bits/weight to Q3_K_M — 3.5 bits/weight. |
| Context length | 8192 tokens | The KV cache grows linearly with this. It is the biggest lever you have. |
Real answers across your hardware
The same calculation run at a range of settings, with everything else left at its default. These are computed by the tool itself, not written by hand.
| Your hardware | Sizes that fit NVIDIA B200 | Usable memory | Quantisation | Context |
|---|---|---|---|---|
| NVIDIA B200 · 180 GB | 9 of 10 | 165.6 GB | Q4_K_M | 8,192 tokens |
| Cerebras WSE-3 · 44 GB on-chip SRAM | 7 of 10 | 44.0 GB | Q4_K_M | 8,192 tokens |
| Raspberry Pi 5 · 16 GB | 4 of 10 | 9.60 GB | Q4_K_M | 8,192 tokens |
| RTX 4080 SUPER · 16 GB | 5 of 10 | 14.7 GB | Q4_K_M | 8,192 tokens |
| RTX 5070 · 12 GB | 5 of 10 | 11.0 GB | Q4_K_M | 8,192 tokens |
At Q4_K_M and 8,192 tokens of context, NVIDIA B200 holds 9 of the 10 size classes. Speed falls as size rises because each token streams more weights.
How this is calculated
No lookup tables and no invented constants. Here is the arithmetic, so you can check it against your own numbers.
Memory for a model is three things added together: the weights, which are parameters × bits-per-weight ÷ 8; the KV cache, which grows linearly with context length; and about a gigabyte of runtime overhead for the CUDA context, activations and framework.
Capacity follows a model’s total parameter count even for mixture-of-experts designs, because the router may select any expert on the next token and all of them must stay resident. Only throughput follows the active count.
Token generation is memory-bandwidth bound rather than compute bound: each token streams the active weights once. Throughput is therefore memory bandwidth divided by the bytes read per token, derated 0.8 for real-world efficiency. Every device here carries its published bandwidth figure.
These are well-founded engineering estimates, not benchmark results. Your quantisation, runtime and context length all move the real number, and usable memory is an assumption rather than a specification. See the full methodology for every assumption behind these figures.
Who this is for
The three situations that bring people to this calculation.
Choosing a size
See the whole ladder against one card.
Explaining a decision
A table you can show someone else.
Planning
Spot where your hardware runs out.
How to use this calculator
Four steps, no account, nothing leaves your browser.
Set your hardware
Start at the top of the panel. Every figure recalculates as you change it — there is no submit button, because watching the number move is the point.
Adjust the rest to match your setup
2 further settings: quantisation, context length. Defaults are the common case, so change only what differs for you.
Read the headline, then the table
The large figure answers the question. The table underneath shows how the answer changes across nearby settings, which is usually where the decision actually gets made.
Check it against the method
The arithmetic is written out above. If a number looks wrong for your hardware, the assumptions are the first place to look — usable memory and quantisation are the two that vary most.
How do model sizes compare on my hardware: common questions
The questions people ask about this, answered without hedging.
How much better is a 70B than a 7B?
Noticeably better at reasoning, instruction following and long-form work, but not ten times better, and it costs roughly ten times the memory. Recent small models often match older large ones.
What size model do most people run?
7–14B is the common range, because it fits consumer cards at a good quantisation and generates faster than reading speed. 32B is the enthusiast tier; 70B needs 48GB or more.
Is a bigger model always slower?
For dense models, yes and close to proportionally — twice the parameters is twice the bytes read per token. Mixture-of-experts models break this: they run at the speed of their active parameters.
How do I compare models of the same size?
Memory and speed will be nearly identical, so the difference is training. Compare on the benchmarks that match your task rather than on size.
Related local AI tools
All 50 run on the same arithmetic, so answers across them agree.
More on will it fit
Context
Speed
Quantisation
Runtime & errors
Cost
Fine-tuning
Searches this page answers
Different ways of asking the same question, all resolved above.
Now find the models that fit
Sizing is only half the problem. Model Radar takes your hardware and shows which models actually run on it, ranked by what they are good at — the same arithmetic as this page, applied to every model worth running.