Local MoE Inference over the network - viable or dumb?

[kalomaze]

Would it be possible to have a setup with two separate machines, each with a single GPU running on the same local internet connection, in which:

• They both host 4 experts of Mixtral (or any arbritary amount, really) on GPUs. Let's say the "Host" machine hosts Experts 0, 1, 2, 3 while the "Secondary" machine hosts 4, 5, 6, 7.
• At inference time, they would both compute different experts. Each machine only computes the experts that it has access to.
• Both machines (the 'host' and the 'secondary' machine) get transferred the hidden states of the computed layers for each expert over the network.

My thinking is, because you are not transferring a large amount of data (only 16kb for each hidden state), the latency of the network is perhaps not enough of an issue for this to be a bottleneck, especially over local Ethernet with full-duplex communication.

[lin72h]

I'm thinking about a similar architecture. Thank you for pointing out the size of the hidden state. I believe that a low-latency interconnect is more suitable for this application. The latency for InfiniBand (40G, 56G, 100G - HDR, FDR, EDR) is uniformly 600ns within a single subnet (1 hop).