Set Different GPU Memory Utilization for Each GPU

[MohmedMonsef]

I have two GPUs with varying workloads, so one might have more memory than the other. Is there a way to configure vllm to use different amounts of memory for each GPU individually?

I am using

• pipeline_parallel_size: 1
• tensor_parallel_size: 2
  as my two GPUs are located on one node.

And according to my understanding, the same gpu-memory-utilization setting is applied to all GPUs.

[DarkLight1337]

No, this is not possible. It would complicate things too much to support this.

[YangZeyu95]

No, this is not possible. It would complicate things too much to support this.

How about in different nodes? I have 2 L40 servers with 8x48 memory and one 4090 server with 8x24G memory. I tried to run model -tp 8 -pp 3 but got cuda OOO. Is there any way to achieve that? In my understanding, pp splits the model by layer which is easy to support compared with tp.