How fast will this model run on my hardware?
Generation speed is not a mystery: to produce one token the model reads every weight once, so tokens per second is memory bandwidth divided by the size of the weights.
The short answer
Computed from this tool’s default settings — your hardware and the rest as most people start. Change them below for your own case.
About 179 tokens per second — 1,008 GB/s divided by 4.50 GB read per token, derated 0.8. Below roughly 10 tok/s a chat model feels slow to read along with.
Tokens-per-Second Estimator
Bandwidth-bound throughput for any model on any device in the catalogue.
About 179 tokens per second — 1,008 GB/s divided by 4.50 GB read per token, derated 0.8. Below roughly 10 tok/s a chat model feels slow to read along with.
| Device in this tier | Bandwidth | Throughput |
|---|---|---|
| RTX 5090 · 32 GB | 1,792 GB/s | 319 tok/s |
| RTX 5080 · 16 GB | 960 GB/s | 171 tok/s |
| RTX 5070 Ti · 16 GB | 896 GB/s | 159 tok/s |
| RTX 5070 · 12 GB | 672 GB/s | 119 tok/s |
| RTX 5060 Ti 16GB · 16 GB | 448 GB/s | 80 tok/s |
| RTX 4090 · 24 GB | 1,008 GB/s | 179 tok/s |
| RTX 4080 SUPER · 16 GB | 736 GB/s | 131 tok/s |
| RTX 4070 Ti SUPER · 16 GB | 672 GB/s | 119 tok/s |
What each setting changes
Every input moves the result for a reason. This is what each one does and where to find the value for your own machine.
| Setting | Default | What it changes |
|---|---|---|
| Your hardware | RTX 4090 · 24 GB | The machine the model runs on. Usable memory decides what fits and memory bandwidth decides how fast it runs, so this moves every figure below. |
| Model size | 8B — Llama 3.1 8B class | Capacity follows total parameters, even for mixture-of-experts models. |
| Quantisation | Q4_K_M — 4.5 bits/weight | 5 options, from Q8_0 — 8.5 bits/weight to Q3_K_M — 3.5 bits/weight. |
| Active parameters (MoE only) | 0 B | Leave at 0 for a dense model. For mixture-of-experts, this sets the speed. |
Real answers across model size
The same calculation run at a range of settings, with everything else left at its default. These are computed by the tool itself, not written by hand.
| Model size | Estimated throughput | Bytes read per token | Device bandwidth | Reads like |
|---|---|---|---|---|
| 1B — Llama 3.2 1B class | 1,434 tok/s | 0.56 GB | 1,008 GB/s | 1.0B dense |
| 7B — Mistral / Qwen 7B class | 205 tok/s | 3.94 GB | 1,008 GB/s | 7.0B dense |
| 14B — Qwen 14B class | 102 tok/s | 7.88 GB | 1,008 GB/s | 14.0B dense |
| 32B — Qwen 32B class | 44.8 tok/s | 18.0 GB | 1,008 GB/s | 32.0B dense |
| 120B — gpt-oss 120B class | 11.9 tok/s | 67.5 GB | 1,008 GB/s | 120.0B dense |
About 1,434 tokens per second — 1,008 GB/s divided by 0.56 GB read per token, derated 0.8. Below roughly 10 tok/s a chat model feels slow to read along with.
How this is calculated
No lookup tables and no invented constants. Here is the arithmetic, so you can check it against your own numbers.
Token generation is memory-bandwidth bound rather than compute bound: each token streams the active weights once. Throughput is therefore memory bandwidth divided by the bytes read per token, derated 0.8 for real-world efficiency. Every device here carries its published bandwidth figure.
These are well-founded engineering estimates, not benchmark results. Your quantisation, runtime and context length all move the real number, and usable memory is an assumption rather than a specification. See the full methodology for every assumption behind these figures.
Who this is for
The three situations that bring people to this calculation.
Before buying
Predict speed on a card you do not own yet.
Choosing a size
Trade model size against responsiveness.
Sanity checking
Compare what you measure against what the hardware allows.
How to use this calculator
Four steps, no account, nothing leaves your browser.
Set your hardware
Start at the top of the panel. Every figure recalculates as you change it — there is no submit button, because watching the number move is the point.
Adjust the rest to match your setup
3 further settings: model size, quantisation, active parameters (moe only). Defaults are the common case, so change only what differs for you.
Read the headline, then the table
The large figure answers the question. The table underneath shows how the answer changes across nearby settings, which is usually where the decision actually gets made.
Check it against the method
The arithmetic is written out above. If a number looks wrong for your hardware, the assumptions are the first place to look — usable memory and quantisation are the two that vary most.
How fast will this model run on my hardware: common questions
The questions people ask about this, answered without hedging.
Why is memory bandwidth more important than TFLOPs?
Single-stream decoding is memory bound, not compute bound. The card reads the entire weight set for every token and does comparatively little arithmetic with it, so the bandwidth number predicts speed far better than the compute number.
Why is my real speed lower than this estimate?
The estimate applies an 80% efficiency factor, and real runtimes lose more to sampling, attention over a long context and other work on the card. Treat it as a ceiling you approach, not a promise.
Does quantisation make a model faster?
Yes, close to proportionally. Q4 weights are about half the bytes of Q8, so there is half as much to read per token and the model runs close to twice as fast.
Why are MoE models so fast for their size?
Only the active experts are read per token. A 235B model with 22B active reads about 22B worth of weights each step, so it decodes at roughly the speed of a 22B model — while still needing all 235B resident.
Related local AI tools
All 50 run on the same arithmetic, so answers across them agree.
Will it fit
Context
Quantisation
Runtime & errors
Cost
Fine-tuning
Searches this page answers
Different ways of asking the same question, all resolved above.
Now find the models that fit
Sizing is only half the problem. Model Radar takes your hardware and shows which models actually run on it, ranked by what they are good at — the same arithmetic as this page, applied to every model worth running.