How should I size gpu-memory-utilization for vLLM?
vLLM pre-allocates a fraction of the card and fills it with weights plus a paged KV cache. What is left after the weights is what decides how many people it can serve.
The short answer
Computed from this tool’s default settings — your hardware and the rest as most people start. Change them below for your own case.
At 0.90 utilisation, RTX 4090 holds the weights plus about 12.1 GB of KV cache — roughly 8 concurrent sequences at 8,192 tokens each.
vLLM Memory Calculator
Work out the fraction to set, and how many concurrent sequences it buys.
At 0.90 utilisation, RTX 4090 holds the weights plus about 12.1 GB of KV cache — roughly 8 concurrent sequences at 8,192 tokens each.
vllm serve <model> \ --gpu-memory-utilization 0.9 \ --max-model-len 8192 \ --max-num-seqs 8
What each setting changes
Every input moves the result for a reason. This is what each one does and where to find the value for your own machine.
| Setting | Default | What it changes |
|---|---|---|
| Your hardware | RTX 4090 · 24 GB | The machine the model runs on. Usable memory decides what fits and memory bandwidth decides how fast it runs, so this moves every figure below. |
| Model size | 8B | 9 options, from 1B to 405B. |
| Quantisation | Q8_0 — 8.5 bits/weight | vLLM commonly serves 16-bit or 8-bit weights rather than GGUF quants. |
| Max sequence length | 8192 tokens | Anywhere from 1,024 to 131,072 tokens. |
Real answers across model size
The same calculation run at a range of settings, with everything else left at its default. These are computed by the tool itself, not written by hand.
| Model size | Concurrent sequences | Weights | Pool at 0.90 | KV pool |
|---|---|---|---|---|
| 1B | 39 | 1.06 GB | 21.6 GB | 19.5 GB |
| 3B | 20 | 3.19 GB | 21.6 GB | 17.4 GB |
| 7B | 10 | 7.44 GB | 21.6 GB | 13.2 GB |
| 8B | 8 | 8.50 GB | 21.6 GB | 12.1 GB |
| 14B | 3 | 14.9 GB | 21.6 GB | 5.73 GB |
At 0.90 utilisation, RTX 4090 holds the weights plus about 19.5 GB of KV cache — roughly 39 concurrent sequences at 8,192 tokens each.
How this is calculated
No lookup tables and no invented constants. Here is the arithmetic, so you can check it against your own numbers.
vLLM pre-allocates a fraction of the card with --gpu-memory-utilization and fills it with weights plus a paged KV cache. Whatever is left after the weights becomes the cache, and the cache is what determines how many requests can be served at once.
vLLM wants the original weights or an AWQ/GPTQ build rather than GGUF, which is why the default here is a higher-precision setting than the llama.cpp tools use.
These are well-founded engineering estimates, not benchmark results. Your quantisation, runtime and context length all move the real number, and usable memory is an assumption rather than a specification. See the full methodology for every assumption behind these figures.
Who this is for
The three situations that bring people to this calculation.
Deploying vLLM
Set gpu-memory-utilization from arithmetic, not trial and error.
Capacity planning
Turn memory into concurrent sequences.
Debugging startup
Understand why max-model-len is rejected.
How to use this calculator
Four steps, no account, nothing leaves your browser.
Set your hardware
Start at the top of the panel. Every figure recalculates as you change it — there is no submit button, because watching the number move is the point.
Adjust the rest to match your setup
3 further settings: model size, quantisation, max sequence length. Defaults are the common case, so change only what differs for you.
Read the headline, then the table
The large figure answers the question. The table underneath shows how the answer changes across nearby settings, which is usually where the decision actually gets made.
Check it against the method
The arithmetic is written out above. If a number looks wrong for your hardware, the assumptions are the first place to look — usable memory and quantisation are the two that vary most.
How should I size gpu-memory-utilization for vLLM: common questions
The questions people ask about this, answered without hedging.
What should gpu-memory-utilization be?
0.90 is the usual starting point. Higher risks OOM if anything else touches the card; much lower wastes cache and cuts your concurrency for no benefit.
Why does vLLM fail to start with my max-model-len?
Because the weights leave too little of the pool for even one sequence at that length. Lower max-model-len, raise utilisation slightly, or serve a quantised build.
Can vLLM use GGUF files?
GGUF support is limited and not the intended path. vLLM expects the original weights or an AWQ/GPTQ build, which is why the default precision here is higher than the llama.cpp tools use.
What is paged attention?
vLLM stores the KV cache in fixed-size blocks rather than one contiguous slab, so sequences of different lengths pack without waste. It is why vLLM serves more concurrent requests than a naive server on the same card.
Related local AI tools
All 50 run on the same arithmetic, so answers across them agree.
More on runtime & errors
Will it fit
Context
Speed
Quantisation
Cost
Fine-tuning
Searches this page answers
Different ways of asking the same question, all resolved above.
Now find the models that fit
Sizing is only half the problem. Model Radar takes your hardware and shows which models actually run on it, ranked by what they are good at — the same arithmetic as this page, applied to every model worth running.