How does Endpoint communication work in ucx?

[Adahai]

Hello anyone:

there is  key  communication mode in ucx  such as endpoint ，and  a key API in endpoint  such as sendTaggedNonBlocking  that can send massage to the other end。As same,  there also is a key API in the  ucpwork that  the endpoint  belongs to  in use of  receiving massage , such as recvTaggedNonBlocking.
 So, i have some questions about these :
  1.  Why is the API recvTaggedNonBlocking  that receives the message not in the Endpoint According to the common sense？message send by any endpoint that belongs to this woker can be received  by recvTaggedNonBlocking  in same ucpWorker? not one to one,  actually many to one, right?
  2.  Is the OpenMPI protocol encapsulated into the Endpoint communication kernel?  If not, what is the kernel of Endpoint communication? in other words, how is the functionality of Endpoint communication implemented?
  3.  If the message is delivered by GPU to GPU，Whether the message goes through NVLINK or PCIE？If there is no NVLink connection between A and C, but nvlink existed  between A and B, and also existed B and C。Can I modify the Endpoint so that messages go from A to B, and then from B to C in oder to  passed by NVLink all the time?
 
    Look forward to your reply!

[yosefe]

hi,

1. The receive-tag operation can receive messages on any of the endpoints, depending on the tag. it's done this way to match MPI tag matching semantic, which allows receiving from MPI_ANY_SOURCE.
2. send-tag and receive-tag are implementing MPI tag matching semantics. Therefore, OpenMPI library can use these operations to implement MPI_Send and MPI_Recv (and their variants) with little additional logic.
3. If there is NVLINK, it will go through NVLINK. However, currently, there is no capability to use NVLink for GPUs which are not directly connected to each other.

[Adahai]

I got it, However
I would like to further understand how send-tag and receive-tag work, especially how to transfer data between GPUs. Can you provide more information or materials?

I have completed an experiment: CudamemcpypeerAsync copy data from A GPU buffer to B GPU buffer, and then from B GPU buffer to C GPU buffer, and there is no NVLink connection between A and C, but nvlink existed between A and B, and also existed B and C. the result of the experiment is that the transmission bandwidth from A to C by B go through nvlink is much faster than from A to C via PCIE. and no data loss. Therefore , i guess if i can optimize the way of data transmission between GPUs in ucx given that I know the PUG topology information on my machine. Can I achieve it?

[yosefe]

adding @Akshay-Venkatesh

[Akshay-Venkatesh]

@Adahai Even though GPU topology and cuda programming model allows this, there are few reasons we discarded this optimization:

1. There are explicit copies required with the intermediate GPU (B). This would mean that every communication between A and C intended to go over NVLINK through B would require handshakes with B (specifically there are exchanges required with the process owning GPU B for memory handle exchange as well as requesting that process to issue cuMemcpyAsync on behalf of process owning GPU A). From the UCT API viewpoint, considering that cuMemcpyAsync implements one-sided UCT put/get_zcopy operations, this would be non-trivial to implement. It would also have implications on overlap and overall application performance.
2. There's always a possibility that the user sets CUDA_VISIBLE_DEVICES to exclude GPU B so there's a need for some node-level synchronization to get awareness of which GPUs are accessible. Node-level synchronization at the UCT level isn't available yet.
3. With NVswitch availability, there is alltoall NVLINK connectivity between GPUs on many architectures (like DGX-2, DGX-A100) now. This reduces the motivation to implement forwarding approach that you've discussed.