Performance issues with UCX UD transmission when packet size is 4K

[EmmaLee-c]

Performance issues with UCX UD transmission when packet size is 4K

1. Use hpcx-v2.17(ucx1.16)，intel cpu + ndr400G
2. numactl -N 0 ib_send_bw -c UD -d mlx5_0 -a -F
   bytes BW average[MB/sec]
   2 11.17
   4 23.28
   8 46.95
   16 93.43
   32 187.06
   64 374.52
   128 750.02
   256 1500.93
   512 2992.35
   1024 5976.25
   2048 11844.51
   4096 15569.28
3. mpirun -np 2 -H node1:1,node2:1 -x UCX_TLS=ud -x UCX_ZCOPY_THRESH=auto -x UCX_RNDV_THRESH=auto -x UCX_PROTO_ENABLE=y numactl -N 0 osu_bw -m 2:4096
   Size Bandwidth (MB/s)
   2 9.28
   4 19.05
   8 38.43
   16 69.23
   32 138.07
   64 272.90
   128 503.77
   256 857.66
   512 2063.33
   1024 4201.60
   2048 8411.68
   4096 9389.34
4. UCX_TLS=ud numactl -N 0 ucx_perftest -t ucp_am_bw -s 4096 -n 10000000
   bandwidth (MB/s)
   average
   9046.25
   9864.50
   9865.95
   9863.96
   9852.61
5. numactl -N 0 ucx_perftest -t am_bw -d mlx5_0:1 -x ud_verbs -s 4088 -D bcopy -n 10000000
   bandwidth (MB/s)
   average
   10858.72
   10861.54
   10852.11
   10863.82

As can be seen, the performance of osu_bw is relatively low, especially in 4K, and there is no performance improvement when tested with ucx_perf_test. Do you have any optimization methods?
Can ucx_perftest only test one package length at a time? If you want to test 2-4K package lengths, how should I write it?

thank you very much!

[yosefe]

@lijh5 I would guess the issue is memory copy bandwidth , memory copy is needed for MPI at least on the receiver side, and is not needed for the basic ib_send_bw test.

1. What is the bandwidth with RC transport?
2. Can you share the perf top -p <pid> output of both sides, during the 4K message test?

[EmmaLee-c]

1. The maximum bandwidth of RC can reach 386Gb/s, with a 4K packet length bandwidth of 13686.1MB/s

2. When running osu_bw, the sender is zcopy, the receiver：
   25.21% libc-2.31.so [.] __memcpy_avx_unaligned_erms
   19.12% libuct_ib.so.0 [.] uct_ud_mlx5_iface_progress
   13.01% libpthread-2.31.so [.] pthread_spin_lock
   8.79% libuct_ib.so.0 [.] uct_ud_ep_process_rx
   4.54% libucp.so.0 [.] ucp_eager_middle_handler
   4.50% libuct_ib.so.0 [.] uct_ud_mlx5_ep_send_ctl
   3.11% libucp.so.0 [.] ucp_worker_progress
   3.08% libucp.so.0 [.] ucp_eager_first_handler
   3.06% libopen-pal.so.40 [.] opal_progress
   1.68% libucp.so.0 [.] ucp_tag_recv_nbx
   1.08% libucp.so.0 [.] ucp_tag_frag_list_process_queue
   1.07% libuct_ib.so.0 [.] uct_ud_mlx5_iface_post_recv

3. I am using UCX_TLS=ud ucx_perftest -t ucp_am_bw -s 4096 testing alone, Why do there also occur a large number of memory copy operations? If it were UCX_TLS=rc, it wouldn't appear anymore.