How to Achieve Data-Parallel Offline Batch Inference Using vLLM?

[T-Atlas]

I read the documentation and found that vLLM supports tensor parallelism and pipeline parallelism. However, I’ve encountered a scenario where my model can fully fit into the VRAM of a single GPU, and my machine has multiple GPUs available. Is it possible to improve the inference speed for a batch of data by loading the model in parallel across multiple GPUs? I’ve carefully reviewed the documentation but couldn’t find relevant examples. Does vLLM already support this functionality? If not, are there alternative approaches to achieve this? (I’ve tried tensor parallelism, which increases batch throughput but introduces latency overhead, resulting in limited speed improvement.)

[StefanHeng]

I'm in a similar situation. I saw and read this issue.

For me I think it's easier to just write my own multi processing code, one for each vLLM myself.

Also found an official vLLM data parallel example here.