Command to run Mosaic 7B on multi GPU RTX-4090

[karrtikiyer-tw]

I have a RTX 4090 desktop with 2 GPU's, 24 GB each, would the below be the correct way to run the Mosaic 7B inference using VLLM?

CUDA_VISIBLE_DEVICES=0,1 python -m vllm.entrypoints.api_server --model mosaicml/mpt-7b --tensor-parallel-size 2 --trust-remote-code --port 4040 --host 0.0.0.0