How many users can my GPU serve at once?
Weights are shared between requests; the KV cache is not. That single fact sets how many people one card can serve at once.
The short answer
Computed from this tool’s default settings — your hardware and the rest as most people start. Change them below for your own case.
RTX 4090 holds 23 concurrent 4,096-token sessions. Aggregate throughput stays near 179 tok/s while each user sees a share of it.
Concurrent Requests Calculator
Concurrency from KV cache headroom, and what it does to per-user speed.
RTX 4090 holds 23 concurrent 4,096-token sessions. Aggregate throughput stays near 179 tok/s while each user sees a share of it.
| Users | Memory | Per user | Aggregate |
|---|---|---|---|
| 1 | 6.20 GB | 179.2 tok/s | 179 tok/s |
| 2 | 6.89 GB | 89.6 tok/s | 179 tok/s |
| 4 | 8.28 GB | 44.8 tok/s | 179 tok/s |
| 8 | 11.1 GB | 22.4 tok/s | 179 tok/s |
| 16 | 16.6 GB | 11.2 tok/s | 179 tok/s |
What each setting changes
Every input moves the result for a reason. This is what each one does and where to find the value for your own machine.
| Setting | Default | What it changes |
|---|---|---|
| Your hardware | RTX 4090 · 24 GB | The machine the model runs on. Usable memory decides what fits and memory bandwidth decides how fast it runs, so this moves every figure below. |
| Model size | 8B | 9 options, from 1B to 405B. |
| Quantisation | Q4_K_M — 4.5 bits/weight | 5 options, from Q8_0 — 8.5 bits/weight to Q3_K_M — 3.5 bits/weight. |
| Context per user | 4096 tokens | Anywhere from 512 to 32,768 tokens. |
Real answers across model size
The same calculation run at a range of settings, with everything else left at its default. These are computed by the tool itself, not written by hand.
| Model size | Concurrent users | Weights (shared) | Cache per user | Spare for cache |
|---|---|---|---|---|
| 1B | 83 | 0.56 GB | 0.25 GB | 20.5 GB |
| 3B | 45 | 1.69 GB | 0.43 GB | 19.4 GB |
| 7B | 26 | 3.94 GB | 0.65 GB | 17.1 GB |
| 8B | 23 | 4.50 GB | 0.70 GB | 16.6 GB |
| 14B | 14 | 7.88 GB | 0.92 GB | 13.2 GB |
RTX 4090 holds 83 concurrent 4,096-token sessions. Aggregate throughput stays near 1,434 tok/s while each user sees a share of it.
How this is calculated
No lookup tables and no invented constants. Here is the arithmetic, so you can check it against your own numbers.
Each concurrent request needs its own KV cache; the weights are shared. So concurrency is whatever memory remains after the weights, divided by the cache one request consumes.
Bandwidth is shared too. Batched decoding reads the weights once for the whole batch, so total throughput rises with concurrency while per-user speed falls. The per-user figure below divides the single-stream rate by the batch.
These are well-founded engineering estimates, not benchmark results. Your quantisation, runtime and context length all move the real number, and usable memory is an assumption rather than a specification. See the full methodology for every assumption behind these figures.
Who this is for
The three situations that bring people to this calculation.
Serving a team
Check one card covers your peak.
Setting limits
Pick a max-num-seqs that will not thrash.
Sizing an upgrade
See what more memory buys in users.
How to use this calculator
Four steps, no account, nothing leaves your browser.
Set your hardware
Start at the top of the panel. Every figure recalculates as you change it — there is no submit button, because watching the number move is the point.
Adjust the rest to match your setup
3 further settings: model size, quantisation, context per user. Defaults are the common case, so change only what differs for you.
Read the headline, then the table
The large figure answers the question. The table underneath shows how the answer changes across nearby settings, which is usually where the decision actually gets made.
Check it against the method
The arithmetic is written out above. If a number looks wrong for your hardware, the assumptions are the first place to look — usable memory and quantisation are the two that vary most.
How many users can my GPU serve at once: common questions
The questions people ask about this, answered without hedging.
How many users can one GPU serve?
However many KV caches fit after the weights. A 24GB card running an 8B model at Q4 has room for a good number of short sessions, but long contexts cut that quickly.
Does adding users slow everyone down?
Per user, yes; in aggregate, no. Batched decoding reads the weights once for the whole batch, so total throughput stays roughly flat while each user gets a share.
What happens past the limit?
Requests queue rather than fail. That is usually preferable, but queue depth becomes latency, so size for the peak rather than the average.
How do I serve more users on the same card?
Shorten the per-session context, quantise the KV cache, or serve a smaller model. Context length is the biggest lever because cache scales directly with it.
Related local AI tools
All 50 run on the same arithmetic, so answers across them agree.
More on runtime & errors
Will it fit
Context
Speed
Quantisation
Cost
Fine-tuning
Searches this page answers
Different ways of asking the same question, all resolved above.
Now find the models that fit
Sizing is only half the problem. Model Radar takes your hardware and shows which models actually run on it, ranked by what they are good at — the same arithmetic as this page, applied to every model worth running.