Does UCX supprot TCP multi-rail?

[huzhijiang]

Here is the TCP multi-rail test result. It seems UCX always choose one rail to send message, not both. And the perfomance also down a bit if trying to use two TCP devices:

[root@promote ucx-1.15.0]# UCX_TLS=tcp UCX_NET_DEVICES=enp7s0  ucx_perftest 192.168.1.199 -t ucp_am_bw -s 32768 -n 100000
[1710655695.035228] [promote:69998:0]        perftest.c:783  UCX  WARN  CPU affinity is not set (bound to 12 cpus). Performance may be impacted.
+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[thread 0]             27279      2.000    36.659    36.659      852.46     852.46       27279       27279
[thread 0]             55450      2.000    35.498    36.069      880.33     866.40       28171       27725
[thread 0]             83648      2.000    35.464    35.865      881.16     871.32       28197       27882
Final:                100000      2.000    35.615    35.824      877.43     872.31       28078       27914


[root@promote ucx-1.15.0]# UCX_TLS=tcp UCX_NET_DEVICES=enp8s0  ucx_perftest 192.168.1.199 -t ucp_am_bw -s 32768 -n 100000
[1710655711.658207] [promote:70004:0]        perftest.c:783  UCX  WARN  CPU affinity is not set (bound to 12 cpus). Performance may be impacted.
+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[thread 0]             26849      6.000    37.248    37.248      838.97     838.97       26847       26847
[thread 0]             54565      2.000    36.084    36.657      866.04     852.51       27713       27280
[thread 0]             82901      2.000    35.293    36.191      885.43     863.48       28334       27631
Final:                100000      2.000    39.720    36.794      786.77     849.32       25177       27178

[root@promote ucx-1.15.0]# UCX_TLS=tcp UCX_NET_DEVICES=enp7s0,enp8s0  ucx_perftest 192.168.1.199 -t ucp_am_bw -s 32768 -n 100000
[1710655666.995142] [promote:69992:0]        perftest.c:783  UCX  WARN  CPU affinity is not set (bound to 12 cpus). Performance may be impacted.
+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[thread 0]             22812      2.000    43.838    43.838      712.85     712.85       22811       22811
[thread 0]             45946      2.000    43.231    43.533      722.86     717.85       23132       22971
[thread 0]             68145      2.000    45.047    44.026      693.71     709.81       22199       22714
[thread 0]             91011      2.000    43.739    43.954      714.47     710.97       22863       22751
Final:                100000      2.000    44.435    43.997      703.27     710.27       22505       22729

[huzhijiang]

Checked the related code, seems I must construct data in a way to trigger both am_zcopy_first and am_zcopy_middle to drive more than one TCP ifaces to work. So I added the following parameters to ucx_perftest and try again:
-D iov -s 2048,2048,2048,2048,2048,2048,2048,2048,2048,2048,2048,2048,2048,2048,2048,2048
By checking profiling result, now am_zcopy_middle and am_zcopy_first are both triggered. But according to ifconfig from the sender side, still only one NIC sending. Any idea?

[huzhijiang]

The two UCT eps created during wireup seems has wrong peer IP. as below:
`tcp_cm.c:104 UCX DEBUG tcp_ep 0x249cd30: WAITING_ACK -> CONNECTED for the [192.168.200.2:45105]<->[192.168.100.1:45015]:0 connection [Tx:Rx]

tcp_cm.c:104 UCX DEBUG tcp_ep 0x249ce20: WAITING_ACK -> CONNECTED for the [192.168.100.2:53915]<->[192.168.200.1:53281]:0 connection [Tx:Rx]

`

Is that the case?
P.S., UCX version is 1.15.0

[huzhijiang]

from lsof -Pn -p, I can see ucx_perf build two sockets to remote node but use the same local and remote ip port , it seems a issue?

ucx_perft 3264 root 15u IPv4 60939 0t0 TCP 192.168.200.2:37768->192.168.200.1:57119 (ESTABLISHED)
ucx_perft 3264 root 16u IPv4 53571 0t0 TCP 192.168.200.2:46983->192.168.200.1:37442 (ESTABLISHED)

I would like to promote this disscusion to an issue for more attention, many thanks!

[evgeny-leksikov]

@huzhijiang try to set UCX_MAX_EAGER_RAILS=2
there is an example:

$ ucx_info -c | grep EAGER
UCX_MAX_EAGER_RAILS=1

$ UCX_TLS=tcp UCX_NET_DEVICES=ib2,ib3 taskset -c 1 ucx_perftest <IP> -t ucp_am_bw
 -s 32768 -n 100000       
+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[thread 0]             37352      0.667    26.774    26.774     1167.19    1167.19       37350       37350
[thread 0]             72972      0.911    28.075    27.409     1113.10    1140.14       35619       36485
Final:                100000      0.671    26.863    27.261     1163.33    1146.32       37227       36682

$ UCX_TLS=tcp UCX_MAX_EAGER_RAILS=2 UCX_NET_DEVICES=ib2,ib3 taskset -c 1 ucx_perftest <IP> -t ucp_am_bw 
-s 32768 -n 100000
+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[thread 0]             73817      0.078    13.548    13.548     2306.59    2306.59       73811       73811
Final:                100000      0.078    11.161    12.923     2799.90    2418.14       89597       77381

[evgeny-leksikov]

@huzhijiang does UCX_MAX_EAGER_RAILS=2 help?

[huzhijiang]

Hi @evgeny-leksikov ,
UCX_MAX_EAGER_RAILS=2 doesn't help, and I already using it. Please move to issue: #9763 , seems I found the publem.