single node multi-gpu inference - batch processing

[innat-asj]

In the doc, it is mentioned that

To run multi-GPU inference with the LLM class, set the tensor_parallel_size argument to the number of GPUs you want to use. For example, to run inference on 4 GPUs:

from vllm import LLM
llm = LLM("facebook/opt-13b", tensor_parallel_size=4)
output = llm.generate("San Franciso is a")

If I'm not mistaken, using tp=4, the layers of the model will be distributed across 4 GPU. So, eventually, this would be **model-parallel` inferencing. But having a model that could fit into a single GPU, I like to do data-parallel, where each GPU will get a replica of a model. And that will allow me to send 4 input to 4 GPU. Is it possible with vllm? I probably can use python multiprocessing but how to control GPU assignment?

[Jae-1409]

If you want to run vllm with data-parallel, use nginx as a load balancer (check 👉 vllm docs)

BTW, if you are running vllm with one-node server, It is better to use tensor-parallel WRT concurrency and latency (only if your gpus can communicate with each other fast)