How many GPUs does my team need?
Sizing a shared deployment has two constraints — memory for the caches and bandwidth for the throughput — and you need whichever is larger, at peak rather than average.
The short answer
Computed from this tool’s default settings — people on the team and the rest as most people start. Change them below for your own case.
20 people at 20% peak concurrency means 4 active sessions, and each RTX 4090 handles 11 — so 1 card. throughput is the binding constraint, so a higher-bandwidth card helps more than a larger one.
Team GPU Sizing Calculator
Cards required to serve N people, from concurrency and daily volume.
20 people at 20% peak concurrency means 4 active sessions, and each RTX 4090 handles 11 — so 1 card. throughput is the binding constraint, so a higher-bandwidth card helps more than a larger one.
| Constraint | Limit | Why |
|---|---|---|
| Memory limit | 11 sessions/card | 16.6 GB spare after weights |
| Throughput limit | 11 sessions/card | 179 tok/s shared, 15 tok/s each |
| Binding constraint | Throughput | This is what to fix first |
What each setting changes
Every input moves the result for a reason. This is what each one does and where to find the value for your own machine.
| Setting | Default | What it changes |
|---|---|---|
| People on the team | 20 | Your own figure, starting from 20. Change it to match what you actually run. |
| Peak concurrent share | 20 % | How many are actively generating at the busiest moment. |
| Model size | 8B | 6 options, from 3B to 405B. |
| Quantisation | Q4_K_M — 4.5 bits/weight | 5 options, from Q8_0 — 8.5 bits/weight to Q3_K_M — 3.5 bits/weight. |
| Context per session | 8192 tokens | Anywhere from 1,024 to 65,536 tokens. |
| Card to deploy | RTX 4090 · 24 GB | The machine the model runs on. Usable memory decides what fits and memory bandwidth decides how fast it runs, so this moves every figure below. |
Real answers across model size
The same calculation run at a range of settings, with everything else left at its default. These are computed by the tool itself, not written by hand.
| Model size | Cards needed | Concurrent at peak | Sessions per card | Binding constraint |
|---|---|---|---|---|
| 3B | 1 | 4 | 22 | Memory |
| 8B | 1 | 4 | 11 | Throughput |
| 14B | 1 | 4 | 6 | Throughput |
| 32B | 4 | 4 | 1 | Memory |
| 70B | Model does not fit | 4 | 0 | Memory |
20 people at 20% peak concurrency means 4 active sessions, and each RTX 4090 handles 22 — so 1 card. memory is the binding constraint, so a shorter context per session buys you more headroom than a faster card.
How this is calculated
No lookup tables and no invented constants. Here is the arithmetic, so you can check it against your own numbers.
Two constraints decide the count, and you need whichever is larger. Memory: each concurrent session needs its own KV cache while the weights are shared. Throughput: batched decoding shares bandwidth, so per-user speed falls as concurrency rises, and there is a floor below which the service feels broken.
The tool sizes for 15 tokens per second per user at peak, which is around reading speed. Sizing for the average rather than the peak is the usual way these deployments end up feeling slow.
These are well-founded engineering estimates, not benchmark results. Your quantisation, runtime and context length all move the real number, and usable memory is an assumption rather than a specification. See the full methodology for every assumption behind these figures.
Who this is for
The three situations that bring people to this calculation.
Internal deployment
Size hardware for a team before buying.
Budgeting
Turn headcount into a hardware cost.
Deployment feels slow
Find which constraint you actually hit.
How to use this calculator
Four steps, no account, nothing leaves your browser.
Set people on the team
Start at the top of the panel. Every figure recalculates as you change it — there is no submit button, because watching the number move is the point.
Adjust the rest to match your setup
5 further settings: peak concurrent share, model size, quantisation, context per session, card to deploy. Defaults are the common case, so change only what differs for you.
Read the headline, then the table
The large figure answers the question. The table underneath shows how the answer changes across nearby settings, which is usually where the decision actually gets made.
Check it against the method
The arithmetic is written out above. If a number looks wrong for your hardware, the assumptions are the first place to look — usable memory and quantisation are the two that vary most.
How many GPUs does my team need: common questions
The questions people ask about this, answered without hedging.
How many GPUs does a team of 20 need?
It depends on peak concurrency, not headcount. Twenty people typically means two to four generating at once, which one capable card often covers for a small model.
Should I size for average or peak?
Peak. Sizing for the average is the usual reason a deployment feels slow — everyone arrives at the same times of day.
Memory or throughput — which usually binds?
Memory on long contexts, throughput on short ones. The tool names which, because the fix differs: shorter context for memory, a faster card for throughput.
Is one large GPU better than several small ones?
For serving, several smaller cards each running a full copy is often simpler and more robust than one large card — no tensor parallelism, and a failure takes out part of the capacity rather than all of it.
Related local AI tools
All 50 run on the same arithmetic, so answers across them agree.
More on runtime & errors
Will it fit
Context
Speed
Quantisation
Cost
Fine-tuning
Searches this page answers
Different ways of asking the same question, all resolved above.
Now find the models that fit
Sizing is only half the problem. Model Radar takes your hardware and shows which models actually run on it, ranked by what they are good at — the same arithmetic as this page, applied to every model worth running.