test-backend-ops | tensor row parallel examples

[mjkpolo]

Hello,

I was wondering if there is a simple way to use test-backend-ops.cpp with tensor row parallelism (-sm row). Is llama.cpp the only code that shows how to implement tensor parallelism?

Thanks!

[slaren]

test-backend-ops does not support tensor parallelism. As far as I know yes, only llama.cpp supports it. However, all you need to do use it is to allocate the weights in a ggml_backend_cuda_split_buffer_type.

[mjkpolo]

thank you! I will try this much appreciated 😎