How to know how much real GPU memory is used?

[ikalista]

I know vllm comes up with a controlled area to store KV cached, but a lot of it is actually not really used. May I know that is there any way to measure real GPU memory usage?

[juni3227]

If you are using Nvidia, execute nvidia-smi in the other terminal, look for vllm instance. For AMD, I do not know.

[chenxu2048]

vLLM records cache usage, logs them and exposes them via prometheus. We can also recalculate the GPU memory usage from GPU block numbers and it usage. But here are not direct GPU memory usage by kvcache in vLLM for now.

INFO 03-14 11:50:43 llm_engine.py:338] # GPU blocks: 2236, # CPU blocks: 655
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO:     Started server process [112564]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:1758 (Press CTRL+C to quit)
INFO 03-14 11:50:58 metrics.py:205] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%
INFO 03-14 11:51:08 metrics.py:205] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%

The latest code also logs how many GPU memory used while loading model weights.

INFO 03-11 13:19:13 model_runner.py:96] Loading model weights took 14.3919 GB

You can also estimate the GPU memory usage by (total_gpu_memory * gpu_memory_utilization - model_weight_usage) * gpu_kv_cache_usage approximately.

[AI-jitsu]

Do you have an example code to get the metric gpu_kv_cache_usage and any other metric from the logs ? I would like to get these metrics used as variables instead of copying them from the logs

[chenxu2048]

https://docs.vllm.ai/en/latest/serving/metrics.html.

You can use these metrics.

[googlercolin]

I would like to calculate the usage of the KV cache in GiB, however, I am not very sure if num_total_gpu = self.cache_config.num_gpu_blocks in the code means the total number of blocks allocated for the KV cache (6.38GiB as per my logging output below) or the total number of GPU blocks (39.50GiB)?

Also, when scheduler.block_manager.get_num_free_gpu_blocks() is called, are the free blocks only taken from those allocated to for the KV cache (part of the 6.38GiB), or it includes the 0.10 (3.95 GiB) outside the gpu_memory_utilization?

Logs:

INFO 02-24 07:59:56 model_runner.py:1115] Loading model weights took 27.5915 GB
INFO 02-24 07:59:58 worker.py:266] Memory profiling takes 1.82 seconds
INFO 02-24 07:59:58 worker.py:266] the current vLLM instance can use total_gpu_memory (39.50GiB) x gpu_memory_utilization (0.90) = 35.55GiB
INFO 02-24 07:59:58 worker.py:266] model weights take 27.59GiB; non_torch_memory takes 0.09GiB; PyTorch activation peak memory takes 1.48GiB; the rest of the memory reserved for KV Cache is 6.38GiB.

Thank you!

[chenxu2048]

I would like to calculate the usage of the KV cache in GiB, however, I am not very sure if num_total_gpu = self.cache_config.num_gpu_blocks in the code means the total number of blocks allocated for the KV cache (6.38GiB as per my logging output below) or the total number of GPU blocks (39.50GiB)?

Also, when scheduler.block_manager.get_num_free_gpu_blocks() is called, are the free blocks only taken from those allocated to for the KV cache (part of the 6.38GiB), or it includes the 0.10 (3.95 GiB) outside the gpu_memory_utilization?

Logs:

INFO 02-24 07:59:56 model_runner.py:1115] Loading model weights took 27.5915 GB
INFO 02-24 07:59:58 worker.py:266] Memory profiling takes 1.82 seconds
INFO 02-24 07:59:58 worker.py:266] the current vLLM instance can use total_gpu_memory (39.50GiB) x gpu_memory_utilization (0.90) = 35.55GiB
INFO 02-24 07:59:58 worker.py:266] model weights take 27.59GiB; non_torch_memory takes 0.09GiB; PyTorch activation peak memory takes 1.48GiB; the rest of the memory reserved for KV Cache is 6.38GiB.

Thank you!

@googlercolin

Q1: means total number of blocks allocated for the KV cache.
Q2: 3.95 GiB is not used by kvcache, but may be used by pytorch when inferencing.
vLLM uses 35.55GiB GPU memory and splits it into model weights (27.59GiB), kv cache (6.38GiB) and others.