Why use tensor parallelism when model can easily fit on a single GPU ?

[vikigenius]

If the model can fit on a single GPU, wouldn't it be better to use something like DDP instead? What are the advantages of using tensor parallelism if the model is small enough to fit on a single GPU ?

[zhuohan123]

You are right, for small models, you should just use one GPU. You can start multiple vLLM replicas to achieve "data parallelism" for serving, so that is not shown in our code. Tensor parallelism is mainly for large models that cannot fit a single GPU.

[nivibilla]

How would I go about doing this? Does something as simple as torch.cuda.set_device() work? Can I have 8 processes with each one on a different gpu?

[gesanqiu]

Just start a vLLM server(Engine) on each GPU will work fine.

[pri2si17-1997]

Hi @gesanqiu do you have any code example related to starting vllm replicas? That would be helpful. I am trying but it doesn't seem to work.

[pgarec]

Model parallelism in serving scenarios can be indeed beneficial when serving multiple models at the same time (https://arxiv.org/abs/2302.11665), but in case of serving a single model then you should stick to one GPU