How much faster is speculative decoding?
A small draft model guesses ahead and the large model checks its work in one pass. When the two agree often, you get the big model’s output at a fraction of the wait.
The short answer
Computed from this tool’s default settings — model size and the rest as most people start. Change them below for your own case.
At 70% acceptance and 5 draft tokens, expect about 1.81× — 324 tok/s instead of 179. Output is identical to running the target alone; speculative decoding changes speed, not results.
Speculative Decoding Speedup
Expected speedup from a draft model, at your acceptance rate.
At 70% acceptance and 5 draft tokens, expect about 1.81× — 324 tok/s instead of 179. Output is identical to running the target alone; speculative decoding changes speed, not results.
| Acceptance | Tokens/round | Speedup | Result |
|---|---|---|---|
| 30% | 1.43 | 0.88× | 157 tok/s |
| 50% | 1.97 | 1.21× | 217 tok/s |
| 60% | 2.38 | 1.47× | 263 tok/s |
| 70% | 2.94 | 1.81× | 324 tok/s |
| 80% | 3.69 | 2.27× | 407 tok/s |
| 90% | 4.69 | 2.88× | 517 tok/s |
What each setting changes
Every input moves the result for a reason. This is what each one does and where to find the value for your own machine.
| Setting | Default | What it changes |
|---|---|---|
| Model size | 8B | 6 options, from 3B to 405B. |
| Draft model size | 1B | The draft must share the target’s tokeniser. |
| Draft length | 5 tokens ahead | Anywhere from 1 to 12 tokens ahead. |
| Acceptance rate | 70 % | How often the target agrees with the draft. 60–80% is typical for a well-matched pair. |
| Your hardware | RTX 4090 · 24 GB | The machine the model runs on. Usable memory decides what fits and memory bandwidth decides how fast it runs, so this moves every figure below. |
Real answers across model size
The same calculation run at a range of settings, with everything else left at its default. These are computed by the tool itself, not written by hand.
| Model size | Speedup | Without | With | Tokens per round |
|---|---|---|---|---|
| 3B | 1.10× | 478 tok/s | 527 tok/s | 2.94 |
| 8B | 1.81× | 179 tok/s | 324 tok/s | 2.94 |
| 14B | 2.17× | 102 tok/s | 222 tok/s | 2.94 |
| 32B | 2.54× | 45 tok/s | 114 tok/s | 2.94 |
| 70B | 2.75× | 20 tok/s | 56 tok/s | 2.94 |
At 70% acceptance the draft is not paying for itself. Use a smaller draft, shorten the draft length, or pick a draft model from the same family as the target so the two agree more often.
How this is calculated
No lookup tables and no invented constants. Here is the arithmetic, so you can check it against your own numbers.
A small draft model proposes k tokens; the large model verifies them all in one pass and keeps the prefix it agrees with. Expected tokens per round is the geometric series (1 − α^(k+1)) ÷ (1 − α), which is the standard result from the speculative-decoding literature.
Speedup is that expectation divided by the cost of a round — one target pass plus k draft passes. Because the draft is much cheaper, the ratio can exceed two, but only when acceptance is high. A poorly matched draft makes things slower, and the tool will say so.
These are well-founded engineering estimates, not benchmark results. Your quantisation, runtime and context length all move the real number, and usable memory is an assumption rather than a specification. See the full methodology for every assumption behind these figures.
Who this is for
The three situations that bring people to this calculation.
Speeding up a large model
See whether a draft pays for itself.
Choosing a draft
Find the size that helps rather than hurts.
Tuning
Balance draft length against acceptance.
How to use this calculator
Four steps, no account, nothing leaves your browser.
Set model size
Start at the top of the panel. Every figure recalculates as you change it — there is no submit button, because watching the number move is the point.
Adjust the rest to match your setup
4 further settings: draft model size, draft length, acceptance rate, your hardware. Defaults are the common case, so change only what differs for you.
Read the headline, then the table
The large figure answers the question. The table underneath shows how the answer changes across nearby settings, which is usually where the decision actually gets made.
Check it against the method
The arithmetic is written out above. If a number looks wrong for your hardware, the assumptions are the first place to look — usable memory and quantisation are the two that vary most.
How much faster is speculative decoding: common questions
The questions people ask about this, answered without hedging.
What is speculative decoding?
A small model proposes several tokens; the large model verifies them all in a single forward pass and keeps the prefix it agrees with. Output is identical to running the large model alone — only the speed changes.
How much faster does it get?
Commonly 1.5–2.5× with a well-matched draft and a high acceptance rate. A poorly matched draft can make things slower, since the drafting work is wasted when rejected.
Which draft model should I use?
The smallest one from the same family, sharing the tokeniser — a 1B drafting for an 8B, or an 8B drafting for a 70B. Same-family pairs agree far more often.
Does it change the output?
No. The verification step guarantees the same distribution as the target model alone, which is what makes the technique safe to enable by default.
Related local AI tools
All 50 run on the same arithmetic, so answers across them agree.
Will it fit
Context
Quantisation
Runtime & errors
Cost
Fine-tuning
Searches this page answers
Different ways of asking the same question, all resolved above.
Now find the models that fit
Sizing is only half the problem. Model Radar takes your hardware and shows which models actually run on it, ranked by what they are good at — the same arithmetic as this page, applied to every model worth running.