How can I increase speed when using multiple smartphones?

[myan-o]

I tried using the RPC function, but it ended up being slower. Is there something wrong with the split approach?

[ggerganov]

The RPC backend cannot increase the speed - it only distributes the model across multiple devices but evaluates it sequentially

[myan-o]

example

<layer1(input:input_ids)>
-node1(input:input_ids,output:node1_out)
<layer2(input:node1_out)>
-node2(input:node1_out,output:node2_out)
-node3(input:node1_out,output:node3_out)
-node4(input:node1_out,output:node4_out)

If the node1_out variable a has already been processed, node2/node3/node4 can be processed simultaneously, right?

[myan-o]

Can parts that are executed in parallel using openmp be distributed using RPC?

[ggerganov]

example

Seems like you are referring to Tensor Parallelism over the network. This is not supported currently

Can parts that are executed in parallel using openmp be distributed using RPC?

It has to be implemented at a higher level

[myan-o]

I think it would be easy to implement if you know the code in llama.cpp, but I don't know it, so it's going to be a pain.

[myan-o]

Would it be possible to split cblas_sgemm in ggml_blas.c and perform distributed computing?