Getting "Connection refused" error while setting up a cluster of 2 nodes

[shurikasa]

Hello guys,

Trying to set up and run PRESTO code on a centos 2 nodes cluster to run sql-type queries I am getting the following "Connection refused" errors:
[1616692472.672278] [gabe-bm-95:18810:1] sock.c:261 UCX ERROR connect(fd=2104, dest_addr=127.0.0.1:53996) failed: Connection refused
2021-03-26T01:14:33.921+0800 INFO pool-61-thread-1 nova.hetu.shuffle.ucx.UcxFetch GET PAGES DONE UcxFetch{producerId='20210325_171407_00010_bygmr.0.0-0', id=18, closed=true, msgid=2}
2021-03-26T01:14:35.921+0800 INFO pool-61-thread-1 nova.hetu.shuffle.ucx.UcxFetch GET PAGES DONE UcxFetch{producerId='20210325_171411_00011_bygmr.0.0-0', id=24, closed=true, msgid=2}
2021-03-26T01:14:35.921+0800 INFO 20210325_171430_00012_bygmr.10.0-3-99 nova.hetu.shuffle.ucx.UcxConnection Create a new connection pool for gabe-bm-96 Max connections 1
[1616692475.945974] [gabe-bm-95:18810:3] ucp_worker.c:1720 UCX INFO ep_cfg[2]: tag(tcp/enp2s0f0);
[1616692475.949630] [gabe-bm-95:18810:3] ucp_worker.c:1720 UCX INFO ep_cfg[3]: tag(tcp/enp2s0f0 tcp/lo);
[1616692475.954719] [gabe-bm-95:18810:3] sock.c:261 UCX ERROR connect(fd=2113, dest_addr=127.0.0.1:56203) failed: Connection refused

I am using one connection per host, I have explicitly set UCX_TLS=tcp (although I have tried a lot of combinations). The initial connection is set up and a couple of exchanges between nodes go through, however very quickly there is that "connection refused" error on one or another node. I have disable the firewall.

Please kindly let us know what could potentially be the issue. BTW no more information is available, the UCX_LOG_LEVEL is set to TRACE. Thanks a lot!

[alinask]

Hi @shurikasa,
A few questions please:
Which UCX version are you using?
Are you setting any other env parameters besides UCX_TLS?
Can you please share the output of both sides with UCX_LOG_LEVEL=TRACE? (need to recompile UCX with debug mode (contrib/configure-devel))
Do you have a stable reproducer that you can share?
Thanks!

[shurikasa]

Hi @alinask,
Thanks a lot for your prompt reply.
I'm using UCX 1.10.0. Here are other parameters I'm using (tried to use sm,tcp in UCX_TLS):
export http_proxy=http://127.0.0.1:3128
export https_proxy=http://127.0.0.1:3128
export ftp_proxy=http://127.0.0.1:3128
ulimit -c unlimited
export UCX_TLS=sm,tcp
export UCX_LOG_LEVEL=TRACE
sysctl -w net.ipv4.tcp_max_syn_backlog=4096
sysctl -w net.core.somaxconn=15000
sysctl -w net.core.netdev_max_backlog=1000

I've read in one of the issues related to the UCX that setting net.ipv4... etc. parameters might resolve the "Connection refused" issue. As for the f/http*_proxy parameters - those are needed for the ethernet setup, but I guess they are not conflicting.

Please allow me to paste the logs corresponding to each node via files since those are bigger to just copy-paste. I've recorded all the trace messages starting from the connection setup. I was trying to run just one single query where I'm asking how many nodes are in the system. I believe this scenario is reproducible, i.e. node 2 always has the connection refused.

Thanks a lot for your help!!
node2.txt
node1.txt

[yosefe]

@shurikasa can you pls try to revert the PR #6198 in your local copy of UCX code (git revert 8e63b6dd) to see if it helps resolve the issue?

[shurikasa]

Hi @yosefe,

Thank you for your suggestion, however that didn't work as well. I'm attaching the log files again, but probably it won't be that useful since it's an older version of the code and the ultimate error is still the same.

Thanks a lot for your help, guys, appreciate that!
node2.txt
node1.txt

[yosefe]

Hi @shurikasa ,

Thank you for your suggestion, however that didn't work as well.

i don't see any error in these log files. Can you please share the full output of the job (with the reverted commit)?

[alinask]

Hi @shurikasa ,
Can you please test your application with the UCX master branch?
We recently merged in a fix there that might fix your issue.
Thanks,
Alina.