Performance issue with 4K packet length

[EmmaLee-c]

The test command is as follows:

1. use tag_bw
   UCX_TLS=rc UCX_NET_DEVICES=mlx5_0:1 UCX_ZCOPY_THRESH=16384 UCX_RNDV_THRESH=16384 numactl -N 0 ucx_perftest -t tag_bw -s 4088
   +--------------+--------------+------------------------------+---------------------+-----------------------+
   | | | overhead (usec) | bandwidth (MB/s) | message rate (msg/s) |
   +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
   | Stage | # iterations | 50.0%ile | average | overall | average | overall | average | overall |
   +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
   [thread 0] 1596961 0.230 0.626 0.626 6224.36 6224.36 1596554 1596554
   [thread 0] 3188257 0.230 0.629 0.627 6203.02 6213.69 1591081 1593818
   [thread 0] 4785185 0.230 0.626 0.627 6224.57 6217.32 1596607 1594747
   [thread 0] 6381601 0.230 0.626 0.627 6223.32 6218.82 1596287 1595132
   [thread 0] 7979553 0.230 0.626 0.627 6229.42 6220.94 1597851 1595676
   [thread 0] 9576481 0.230 0.626 0.627 6224.44 6221.52 1596575 1595826
   Final: 10000000 0.230 0.626 0.627 6229.85 6221.87 1597961 1595916

2. use ucp_am_bw
   UCX_TLS=rc UCX_NET_DEVICES=mlx5_0:1 UCX_MAX_EAGER_LANES=4 UCX_MAX_RNDV_LANES=4 UCX_ZCOPY_THRESH=16384 UCX_RNDV_THRESH=16384 numactl -N 0 ucx_perftest -t ucp_am_bw -s 4088
   +--------------+--------------+------------------------------+---------------------+-----------------------+
   | | | overhead (usec) | bandwidth (MB/s) | message rate (msg/s) |
   +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
   | Stage | # iterations | 50.0%ile | average | overall | average | overall | average | overall |
   +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
   [thread 0] 2340027 0.260 0.427 0.427 9133.24 9133.24 2342684 2342684
   [thread 0] 4999162 0.240 0.376 0.400 10378.73 9755.98 2662154 2502419
   [thread 0] 7635845 0.280 0.379 0.392 10291.09 9934.35 2639674 2548170
   Final: 10000000 0.240 0.378 0.389 10311.26 10020.95 2644847 2570383

3. Question:
   The results of tests tag_bw and ucp_am_bw are significantly different. Can tag_bw adjust any parameters to achieve the performance of ucp_am_bw?
   Because MPI applications, such as osu_bw, use tag matching, will result in very low performance！

Thank you.

[yosefe]

@lijh5 what is the UCX version? Is it possible to run "perf top" on sender and receiver side to see what takes the most CPU time?
The difference can be because with tag_bw there is also memory copy on the receiver side (from UCX buffer to user buffer) which does not happen with ucp_am_bw test.

[EmmaLee-c]

@yosefe 1. use hpcx-v2.14, ucx-v1.15.0
2. perf top result:
tag_bw:
sender:
31.69% libuct_ib.so.0 [.] uct_rc_mlx5_iface_progress_cyclic
20.78% libc-2.28.so [.] __memmove_avx_unaligned_erms
14.95% libucp.so.0 [.] ucp_worker_progress
10.83% ucx_perftest [.] ucp_perf_test_runner<(ucx_perf_cmd_t)7, (ucx_perf_test_type_t)2, 0u>::run_stream_uni
5.08% libuct_ib.so.0 [.] uct_rc_mlx5_ep_am_bcopy
4.88% libucp.so.0 [.] ucp_tag_send_nbx
2.69% libucp.so.0 [.] ucp_tag_eager_bcopy_single
2.51% libucp.so.0 [.] ucp_tag_pack_eager_only_dt
1.31% ucx_perftest [.] ucp_worker_progress@plt
0.86% libucp.so.0 [.] ucp_dt_pack
0.82% libucp.so.0 [.] __popcountdi2
0.61% libuct_ib.so.0 [.] uct_ib_mlx5_txwq_validate_always
0.60% libucp.so.0 [.] ucp_request_send_start
0.53% libucp.so.0 [.] ucp_dt_contig_pack
0.28% libucs.so.0 [.] ucs_mpool_put
0.14% libucs.so.0 [.] ucs_arbiter_dispatch_nonempty
0.07% libuct_ib.so.0 [.] uct_ib_mlx5_txwq_validate_always@plt
0.07% libucp.so.0 [.] ucp_dt_pack@plt

recevier:
64.37% libc-2.28.so [.] __memmove_avx_unaligned_erms
16.12% libuct_ib.so.0.0.0 [.] uct_rc_mlx5_iface_progress_cyclic
3.68% libucp.so.0.0.0 [.] ucp_tag_recv_nbx
3.03% ucx_perftest [.] ucp_perf_test_runner<(ucx_perf_cmd_t)7, (ucx_perf_test_type_t)2, 0u>::run_stream_uni
2.78% libucp.so.0.0.0 [.] ucp_eager_only_handler
0.98% libucp.so.0.0.0 [.] ucp_request_free
0.66% libuct_ib.so.0.0.0 [.] uct_rc_mlx5_iface_srq_post_recv
0.56% libucp.so.0.0.0 [.] ucp_worker_progress

ucp_am_bw:
sender:
44.17% libc-2.28.so [.] __memmove_avx_unaligned_erms
9.19% libuct_ib.so.0 [.] uct_rc_mlx5_ep_am_bcopy
7.16% libucp.so.0 [.] ucp_am_send_nbx
6.80% libucp.so.0 [.] ucp_am_bcopy_single
5.48% ucx_perftest [.] ucp_perf_test_runner<(ucx_perf_cmd_t)0, (ucx_perf_test_type_t)2, 0u>::run_stream_uni
4.96% libuct_ib.so.0 [.] uct_rc_mlx5_iface_progress_cyclic
2.81% libucp.so.0 [.] ucp_am_send_req_init
2.33% libucs.so.0 [.] ucs_arbiter_dispatch_nonempty
1.97% libucp.so.0 [.] ucp_am_bcopy_pack_args_single
1.84% libucp.so.0 [.] ucp_dt_pack
1.46% libucp.so.0 [.] ucp_request_send_start
1.39% libuct_ib.so.0 [.] uct_ib_mlx5_txwq_validate_always
1.09% libucp.so.0 [.] ucp_dt_contig_pack
0.78% libucp.so.0 [.] __popcountdi2
0.75% libucp.so.0 [.] ucp_request_pending_add
0.62% libuct_ib.so.0 [.] uct_rc_ep_process_pending
0.62% libucp.so.0 [.] ucp_request_free
0.57% libucs.so.0 [.] ucs_mpool_put
0.55% libuct_ib.so.0 [.] uct_rc_ep_pending_add

recevier:
70.75% libuct_ib.so.0.0.0 [.] uct_rc_mlx5_iface_progress_cyclic
9.25% libucp.so.0.0.0 [.] ucp_worker_progress
5.14% ucx_perftest [.] ucp_perf_test_runner<(ucx_perf_cmd_t)0, (ucx_perf_test_type_t)2, 0u>::run_stream_uni
4.21% libucp.so.0.0.0 [.] ucp_am_handler
1.64% [kernel] [k] pci_conf1_read
1.50% [kernel] [k] pci_conf1_write
0.92% libuct_ib.so.0.0.0 [.] uct_rc_mlx5_iface_srq_post_recv
0.75% ucx_perftest [.] ucp_worker_progress@plt

From this perspective, it is true that tag_bw has done a lot of mem copying in the reviewer.
This will result in low performance when executing mpirun - np 2 osu_bw - m 4096:4096. Is there any way to optimize it?

[yosefe]

For optimizing sender side copy, can try reducing UCX_ZCOPY_THRESH value, or removing it altogether.
For optimizing receiving side copy (with tag_bw), can try reducing UCX_RNDV_THRESH value.