How many GPUs do I need for this model?
Past a certain size no single card holds the model, and the question becomes how many you need — and whether the count is one tensor parallelism will actually accept.
The short answer
Computed from this tool’s default settings — model size and the rest as most people start. Change them below for your own case.
One RTX 5090 holds a 8B model at Q4_K_M — no splitting required.
Multi-GPU Split Calculator
Cards required at each quantisation, and what that costs.
One RTX 5090 holds a 8B model at Q4_K_M — no splitting required.
| Quantisation | Total needed | Cards | Cost |
|---|---|---|---|
| Q8_0 | 10.9 GB | 1 | $1,999 |
| Q6_K | 8.99 GB | 1 | $1,999 |
| Q5_K_M | 8.09 GB | 1 | $1,999 |
| Q4_K_M | 6.89 GB | 1 | $1,999 |
| Q3_K_M | 5.89 GB | 1 | $1,999 |
What each setting changes
Every input moves the result for a reason. This is what each one does and where to find the value for your own machine.
| Setting | Default | What it changes |
|---|---|---|
| Model size | 8B | 9 options, from 1B to 405B. |
| Card to use | RTX 5090 · 32 GB | The machine the model runs on. Usable memory decides what fits and memory bandwidth decides how fast it runs, so this moves every figure below. |
| Context length | 8192 tokens | Anywhere from 1,024 to 131,072 tokens. |
Real answers across model size
The same calculation run at a range of settings, with everything else left at its default. These are computed by the tool itself, not written by hand.
| Model size | Cards needed at Q4_K_M | Per card | Model needs | Hardware cost |
|---|---|---|---|---|
| 1B | 1 | 29.4 GB | 2.05 GB | $1,999 |
| 3B | 1 | 29.4 GB | 3.54 GB | $1,999 |
| 7B | 1 | 29.4 GB | 6.24 GB | $1,999 |
| 8B | 1 | 29.4 GB | 6.89 GB | $1,999 |
| 14B | 1 | 29.4 GB | 10.7 GB | $1,999 |
One RTX 5090 holds a 1B model at Q4_K_M — no splitting required.
How this is calculated
No lookup tables and no invented constants. Here is the arithmetic, so you can check it against your own numbers.
Tensor-parallel serving splits a model across cards, so the requirement is the total memory divided by what one card offers. In practice you also lose some efficiency to interconnect overhead, and odd card counts are awkward — tensor parallelism generally wants powers of two.
Capacity follows total parameters even for mixture-of-experts models: every expert must be resident somewhere, so sharding spreads them rather than reducing them.
These are well-founded engineering estimates, not benchmark results. Your quantisation, runtime and context length all move the real number, and usable memory is an assumption rather than a specification. See the full methodology for every assumption behind these figures.
Who this is for
The three situations that bring people to this calculation.
Large models
Work out the card count for a 70B or beyond.
Budgeting
Turn card count into hardware cost.
Choosing a card
Compare two cards on how many you would need.
How to use this calculator
Four steps, no account, nothing leaves your browser.
Set model size
Start at the top of the panel. Every figure recalculates as you change it — there is no submit button, because watching the number move is the point.
Adjust the rest to match your setup
2 further settings: card to use, context length. Defaults are the common case, so change only what differs for you.
Read the headline, then the table
The large figure answers the question. The table underneath shows how the answer changes across nearby settings, which is usually where the decision actually gets made.
Check it against the method
The arithmetic is written out above. If a number looks wrong for your hardware, the assumptions are the first place to look — usable memory and quantisation are the two that vary most.
How many GPUs do I need for this model: common questions
The questions people ask about this, answered without hedging.
How many GPUs do I need for a 70B model?
At Q4_K_M a 70B needs roughly 40 GB plus cache, so two 24GB cards or one 48GB card. At 16-bit it is about 140 GB and you are into four cards or more.
Does splitting across GPUs make it faster?
Not much for single-stream generation — you are adding capacity, not bandwidth per token, and interconnect overhead eats some of it. It helps most with concurrency.
Why do people say to use powers of two?
Tensor parallelism divides attention heads across cards, and head counts are powers of two. Three cards often will not shard cleanly even when the memory adds up.
Do the cards need to match?
For tensor parallelism, effectively yes — the slowest and smallest card sets the pace and the per-shard budget. Mixed cards work better with pipeline splitting.
Related local AI tools
All 50 run on the same arithmetic, so answers across them agree.
More on hardware
Will it fit
Context
Speed
Quantisation
Runtime & errors
Cost
Searches this page answers
Different ways of asking the same question, all resolved above.
Now find the models that fit
Sizing is only half the problem. Model Radar takes your hardware and shows which models actually run on it, ranked by what they are good at — the same arithmetic as this page, applied to every model worth running.