sock.c:344 UCX ERROR recv(fd=47) failed: Connection reset by peer

[smallriver666]

I encountered this problem when submitting tasks with multiple nodes. I have a management node and five computing nodes. Nodes 2 to 5 are the same server. Node 6 is not the same server as the other four nodes. I am on 2345 four Tasks can be successfully submitted on nodes, and the following error will occur when node 6 is added (I am using openmpi4.1.0)

This is the script I used to submit the task:

[yosefe]

can you pls add -x UCX_LOG_LEVEL_TRIGGER=error -x UCX_HANDLE_ERRORS=bt,freeze to mirin command, and post the printed backtrace?