ECE development under DevX

[changchengx]

1. ECE has 32 bits in HCA, any document to introduce the per bit feature function?

1. Done: check internal wiki
2. Ask Internal employee.

2. Do we need to consider negotiate the retry_mode in UCX? selective repeative, go-back-n, default

1. If selective_repeat is supported on both side, then it should use "SR"
2. If 1) failed, then it should use "go-back-to-N" if both supports
3. If 1) & 2) failed, then it should use "default" mode.

3. **How to choose the input ECE version?
   Create QP with ECE v1.0, then query the ece to get the supported highest ece version.

4. In ucp_cm_client_connect_cb, the remote qpn and ece info could be got.

1. Could we use the same data path to exchange ece info?
   uct_cm_remote_data_t::conn_priv_data
2. Where is the code to setup this data path and exchange the info, like ece and qpn?
   Exchange ECE information as part of UCT address (When adding ECE info into UCT address, the UCT address must be compatible with other UCX version)
   UCT get iface address (uct_iface_get_address)
   UCT get ep address (uct_ep_get_address)
   UCT get device address (uct_iface_get_device_address)
   The address info will be as part of rdmacm private data to be sent to the remote side. It could be used as part of the connect callback(unpack the address)

Write down the changes in a document

1. focus on which APIs and data structure are changed.
2. how to solve backward compatibility
3. protocol diagram

For DC case:

1. evaluate it work or not.
2. if it works, refine it

[changchengx]

rdmacm follow is below:
35 * ibv_create_qp() <- create data QP.
36 * ece = ibv_get_ece() <- get ECE from libibvers provider.
37 * rdma_set_local_ece(ece) <- set desired ECE options.
38 * rdma_connect() <- send connection request
39 * ece = rdma_get_remote_ece() <- get ECE options from remote peer
40 * ibv_set_ece(ece) <- set local ECE options with data received from the peer.
41 * ibv_modify_qp() <- enable data QP.
42 * rdma_accept()/rdma_establish()/rdma_reject_ece()