Multiple GPUs

[austin2035]

Thank you very much for your work, the inference speed has indeed been dramatically improved and the GPU utilization has been increased by several times.

I'm curious if inference can be further improved by tensor parallelism across multiple GPUs.

[sgsdxzy]

You should be able to use tensor parallelism provided by exl2, but I am not familiar with it.

[DocShotgun]

The current implementation doesn't expose any of the exllamav2 GPU split and TP options, and instead behaves the same way as the original repo where --cuda_idx specifies which GPU to throw everything on. I think it would be technically possible to expose those for stage 1 and stage 2, although it might overcomplicate the CLI args. Since it's a small model, it might be more efficient to just run a separate instance on each GPU with the device index.

[austin2035]

according to exllamav2 documents

I have rewrited some places.

But it doesnt works.

I am not very familiar with deep learning, if anyone has any ideas, you can tell me in the comments section.

[sgsdxzy]

@turboderp any ideas?

[DocShotgun]

I'm not sure if this is the issue, but in your changes, you didn't change the actual value of gpu_split. By default in YuE-exllamav2, it puts everything on a single GPU (GPU0 if not specified).

[austin2035]

If multi-GPU inference is possible, it would be a huge boost for inference speed.

[sgsdxzy]

@austin2035 Possibly not, the models are too small (7B, 2B) to benefit, the overhead might slow it down instead.

[austin2035]

Make sense. Thank you.

[dengtao3rd]

I also encountered the same problem. Can you post your solution? @austin2035 @sgsdxzy @DocShotgun